Greylilac09/fix mem tracking #4
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* naively copy e2e otel-logs to otel-metrics * try to fix * Remove vector_default.yaml custom mapping * Test all metric types * Update vrl to main after fix was merged * Add additional attributes and verify them in tests * Format + remove Summary support * simplify assert_metric_attributes logic * Revert Cargo.toml changes * Add ci e2e trigger * Make test more concise * reuse common Dockerfiles * Use common functions in opentelemetry-logs * Hard code vector_otlp.yaml * Fix CI trigger * Add service to integration.yml matrix * Add and use generic assert_service_name_with * ranem data_type -> test_type --------- Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com>
…v#24115) chore(ci): parallelize e2e tests
…vectordotdev#24028) * Fix problem where multi-chars delimiters fail to be parsed if they happen right at a buffer boundary. * Add changelog entry and remove debug println statements * Update changelog.d/24027_file_source_multi_char_delimiter.fix.md Co-authored-by: Thomas <thomasqueirozb@gmail.com> * Add #[cfg(test)] to test method * Expand comments * Add a regression/integration test to file source * Remove cfg test --------- Co-authored-by: Thomas <thomasqueirozb@gmail.com>
…lization (vectordotdev#24272) * enhancement(sources): Add internal metric to record source buffer utilization * Tweak wording * Rework and expand the metrics output by the channel * Fix and expand tests * Address review feedback and simplify the metric names * Fix missing metric rename
…ev#24317) * chore(deps): bump VRL version to include example location * Update example in dnstap * Add bogus source_file/source_line to vector-vrl docs tests * Use VRL commit before datadog search changes
…tdev#24332) chore: bump maxminddb to 0.27 after RUSTSEC-2025-0132
…#24334) bump vrl and fix datadog search tests
…ectordotdev#24296) * fix for syslog udp metrics * add changelog * add authors * Address review comments - use byte_size directly in SocketEventsReceived metrics and remove SocketBytesReceived * use byte_size directly... * clippy * fmt * add back BytesReceived metrics caught by unit test * Use events.estimated_json_encoded_size_of() * clippy --------- Co-authored-by: Emily Chen <emily.chen@datadoghq.com> Co-authored-by: Thomas <thomas.schneider@datadoghq.com>
…vectordotdev#24313) * chore(observability): Introduce `trait NamedInternalEvent` and derive This change moves the two optional `fn name` methods from the `InternalEvent` and `RegisterInternalEvent` traits into a new `trait NamedInternalEvent` and adds a derive procedural macro to make implementation of that trait trivial. This borrows heavily from vectordotdev#23978 which implemented a `#[internal_event]` procedural macro, but takes it in a different direction and completes the implementation. * Miscellaneous fixes * Drop `DefaultName` wrapper as it is no longer needed * Fix Windows service events
…tion (vectordotdev#24329) * enhancement(transforms): Add internal metric to record buffer utilization This builds on the work in vectordotdev#24272 and adds support for transform buffer utilization. Since these are input buffers they omit the `output` label used by the buffers created by the source senders. * Deduplicate metric creation a bit
…ectordotdev#24351) * bump hyper to 0.14.32 * Bump http-body to 0.4.6 and apply suggestions * Fix clippy * Update LICENSE-3rdparty.csv * cargo fmt --------- Co-authored-by: Benson Fung <fung.benson@gmail.com>
* Use VDEV env var in Makefile * Use mold and cargo-cache if vdev needs to be compiled * Setup vdev in test.yml * Fix incorrect return code from inlined if * Add mold: false to deny.yml
* fix * fix * fix * fix * fix * fix * fix * fix * fix
…tobuf (vectordotdev#24336) The existing code would create an `EventMetadata::default()` and then fill in the fields coming from the protobuf metadata. Unfortunately, this default creates several non-empty values, notably the `source_event_id`, which makes that construct take extra time, particularly since the incoming data will (almost?) always already have a `source_event_id` if the origin is another Vector instance. While refactoring, I also changed the method to use destructuring to access the fields inside the source value to force the compiler to error if any fields are added later.
…ucer fatal "Inconsistent state" stalls (vectordotdev#24197) * chore(deps): update rdkafka to 0.38.0 Signed-off-by: hansedong <skipper1314@gmail.com> * cargo vdev build licenses * Use clang in cross/bootstrap-ubuntu.sh --------- Signed-off-by: hansedong <skipper1314@gmail.com> Co-authored-by: Thomas <thomas.schneider@datadoghq.com>
…ordotdev#24358) * Enable fast-rng feature for UUID generation * Use UUID v4 generation instead of V7 v7 are created using a timestamp so they can be ordered, however this comes at a performance cost. We currently don't need to order these UUID's, so for now we can use v4. * Add regression test for disk buffer v2 implementation * Rename metadata test for UUID v4 * Remove unused uuid v7 feature * Add uuid v7 feature back [Websocket server sink uses it](https://github.com/vectordotdev/vector/blob/72e09673fda9d6fbf933adacea1220bdfae162a8/src/sinks/websocket_server/buffering.rs#L235) for time ordered replays in case the connection drops * Replace source event ID if none when merging event metadata * Update test to describe behavior with UUID v4
) Bumps [actions/checkout](https://github.com/actions/checkout) from 5.0.0 to 6.0.0. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@08c6903...1af3b93) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: 6.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…ordotdev#24289) * feat(aws_s3 source): add S3 download processing duration metric Add histogram metric `s3_object_processing_duration_seconds` to track how long S3 object downloads and processing takes. The metric includes `bucket` and `status` labels to help identify slow buckets and distinguish successful vs failed downloads. The timing captures the full download lifecycle from the GetObject API call through complete ByteStream body consumption. * add author to changelog * refactor: make success/failure different metrics * Move bucket/duration into own variables * derive NamedInternalEvent --------- Co-authored-by: sanjams2 <sanjams2@users.noreply.github.com> Co-authored-by: Thomas <thomas.schneider@datadoghq.com>
….1.1 (vectordotdev#24323) chore(ci): bump aws-actions/configure-aws-credentials Bumps [aws-actions/configure-aws-credentials](https://github.com/aws-actions/configure-aws-credentials) from 5.0.0 to 5.1.1. - [Release notes](https://github.com/aws-actions/configure-aws-credentials/releases) - [Changelog](https://github.com/aws-actions/configure-aws-credentials/blob/main/CHANGELOG.md) - [Commits](aws-actions/configure-aws-credentials@a03048d...61815dc) --- updated-dependencies: - dependency-name: aws-actions/configure-aws-credentials dependency-version: 5.1.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…ectordotdev#24069) * fix(websocket source): reconnect indefinitely when connection fails * chore: add changelog fragment * Update src/common/websocket.rs * Remove references to now deleted fresh_backoff * Fix expect()() * fix: add missing protocol --------- Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com> Co-authored-by: Thomas <thomas.schneider@datadoghq.com>
vectordotdev#24288) * enhancement(codecs): allow configurable null handling in Arrow encoder * chore: update changelog * Remove whitespace from changelog * make fmt * Remove Arc::clone when value can be fully moved --------- Co-authored-by: Thomas <thomasqueirozb@gmail.com> Co-authored-by: Thomas <thomas.schneider@datadoghq.com>
…tordotdev#24147) * Revise Azure instance type recommendations Updated Azure recommendations for instance types. * Add new entry to spelling expect.txt Added D8plds_v6 to spelling expect.txt * Try to fix spelling --------- Co-authored-by: Thomas <thomas.schneider@datadoghq.com>
* chore: clean up some `allow` statements * fix fails when running `cargo check --tests --bin vector --no-default-features`
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Vector configuration
How did you test this PR?
Change Type
Is this a breaking change?
Does this PR include user facing changes?
no-changeloglabel to this PR.References
Notes
@vectordotdev/vectorto reach out to us regarding this PR.pre-pushhook, please see this template.make fmtmake check-clippy(if there are failures it's possible some of them can be fixed withmake clippy-fix)make testgit merge origin masterandgit push.Cargo.lock), pleaserun
make build-licensesto regenerate the license inventory and commit the changes (if any). More details here.