Skip to content

Comments

RDKEMW-14300 : Removed Log truncation & added testApp to measure CPU#49

Closed
karuna2git wants to merge 8 commits intodevelopfrom
topic/LogHugeData
Closed

RDKEMW-14300 : Removed Log truncation & added testApp to measure CPU#49
karuna2git wants to merge 8 commits intodevelopfrom
topic/LogHugeData

Conversation

@karuna2git
Copy link
Contributor

Reason for change: Removed Log truncation & added testApp to measure CPU
Test Procedure: Verify huge data
Risks: Low

@karuna2git karuna2git requested a review from a team as a code owner February 20, 2026 00:33
Copilot AI review requested due to automatic review settings February 20, 2026 00:33
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request removes log message truncation logic and adds a CPU performance test application to measure the impact of logging operations. The changes transition from a fixed-size buffer approach (980 bytes) to dynamic buffer allocation, allowing arbitrarily large log messages without truncation.

Changes:

  • Removed manual log truncation logic that split large messages into chunks, replacing it with log4c's built-in log4c_category_vlog function
  • Refactored format handlers into a unified rdk_format_log function that dynamically reallocates buffers when needed
  • Added a new RDKLOG_FORMAT_WITH_TS_TID log format that includes timestamp and thread ID
  • Replaced old test files with a new CPU performance test that measures logging overhead

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 20 comments.

Show a summary per file
File Description
test/rdk_logger_test_main.c Removed old test main file
test/rdk_logger_debug_test.c Removed old debug test file
test/rdk_logger_cpu_test.c Added new CPU performance measurement test with 4 test scenarios
test/Makefile.am Updated to build new CPU test instead of old test program
src/rdk_debug_priv.c Removed truncation logic, refactored format handlers, added dynamic buffer reallocation, and implemented new format with thread ID
log4crc Changed buffer size from 1024 to 0 for dynamic allocation, added new layout configuration
include/rdk_logger.h Added RDKLOG_FORMAT_WITH_TS_TID enum value

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

usleep(10000);

// Test 4: Print INFO where INFO is enabled.
// CPU taken to print the log messagge
Copy link

Copilot AI Feb 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a typo in the comment. "messagge" should be "message".

Copilot uses AI. Check for mistakes.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot open a new pull request to apply changes based on this feedback

Comment on lines 790 to 791
/****************************************************************
* With TimeStamp format with no ending carriage return / line feed
Copy link

Copilot AI Feb 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment is outdated and misleading. The function now adds a newline character via rdk_format_log (see line 731 where "\n" is appended), so it does have an ending line feed. The comment should be updated to reflect the current behavior.

Copilot uses AI. Check for mistakes.
Reason for change: Removed Log truncation & added testApp to measure CPU
Test Procedure: Verify huge data
Risks: Low
Signed-off-by: Karunakaran A <karunakaran_amirthalingam@cable.comcast.com>
Copy link

Copilot AI commented Feb 20, 2026

@karuna2git I've opened a new pull request, #50, to work on those changes. Once the pull request is ready, I'll request review from you.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings February 20, 2026 01:43
karuna2git and others added 3 commits February 19, 2026 20:47
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copy link

Copilot AI commented Feb 20, 2026

@karuna2git I've opened a new pull request, #51, to work on those changes. Once the pull request is ready, I'll request review from you.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings February 20, 2026 02:02
Copy link

Copilot AI commented Feb 20, 2026

@karuna2git I've opened a new pull request, #52, to work on those changes. Once the pull request is ready, I'll request review from you.

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

src/rdk_debug_priv.c:683

  • getModuleName() casts event->evt_category to char* even though it is treated as read-only. This can trigger warnings (cast discards const) depending on log4c’s definition of evt_category. Prefer using const char* throughout (and avoid casting away const) since the function does not modify the category string.
static const char* getModuleName(log4c_logging_event_t* event)
{
    /** Get the last part of the cagetory as "module" */
    char *p= (char *)(event->evt_category);
    if (NULL == p)
    {

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@github-actions
Copy link


Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.


I have read the CLA Document and I hereby sign the CLA


You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot.

Copilot AI review requested due to automatic review settings February 22, 2026 05:25
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 6 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +608 to +621
// Read and verify log content contains both timestamp and thread ID
FILE* logFile = fopen(logFilePath, "r");
ASSERT_NE(logFile, nullptr) << "Failed to open log file for validation";

char line[1024];
bool foundModuleName = false;
if (fgets(line, sizeof(line), logFile)) {
// Check for full module name (LOG.RDK.ROTATION) which should be in detailed format
if (strstr(line, "LOG.RDK.ROTATION") != NULL) {
foundModuleName = true;
}
}
fclose(logFile);
EXPECT_TRUE(foundModuleName) << "Log should contain module name in detailed format";
Copy link

Copilot AI Feb 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The detailed formats now print only the last segment of the category via getModuleName(...) (e.g., ROTATION), not the full LOG.RDK.ROTATION. As written, this assertion will likely fail even when the format is correct. Update the expectation to match the actual detailed layout (or change the layout to include the full category if that’s the intent).

Suggested change
// Read and verify log content contains both timestamp and thread ID
FILE* logFile = fopen(logFilePath, "r");
ASSERT_NE(logFile, nullptr) << "Failed to open log file for validation";
char line[1024];
bool foundModuleName = false;
if (fgets(line, sizeof(line), logFile)) {
// Check for full module name (LOG.RDK.ROTATION) which should be in detailed format
if (strstr(line, "LOG.RDK.ROTATION") != NULL) {
foundModuleName = true;
}
}
fclose(logFile);
EXPECT_TRUE(foundModuleName) << "Log should contain module name in detailed format";
// Read and verify log content contains the module name in detailed format (short module name)
FILE* logFile = fopen(logFilePath, "r");
ASSERT_NE(logFile, nullptr) << "Failed to open log file for validation";
char line[1024];
bool foundModuleName = false;
if (fgets(line, sizeof(line), logFile)) {
// Check for module name as printed by detailed format (last segment of category)
if (strstr(line, "ROTATION") != NULL) {
foundModuleName = true;
}
}
fclose(logFile);
EXPECT_TRUE(foundModuleName) << "Log should contain short module name in detailed format";

Copilot uses AI. Check for mistakes.
struct tm tm;
char timeBuff[COMCAST_DATAED_BUFF_SIZE] = {0};

/** Get the last part of the cagetory as "module" */
Copy link

Copilot AI Feb 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo in comment: "cagetory" should be "category".

Suggested change
/** Get the last part of the cagetory as "module" */
/** Get the last part of the category as "module" */

Copilot uses AI. Check for mistakes.
Comment on lines +17 to 21
<appender name="rdk_console" type="to_console" layout="rdk_detail_without_ts"/>
<appender name="rdk_syslog" type="to_syslog" layout="rdk_with_tid"/>
<appender name="rdk_journal" type="to_journal" layout="rdk_plaintext"/>
<appender name="rdk_printf" type="to_console" layout="rdk_detail_with_ts"/>

Copy link

Copilot AI Feb 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These changes modify default logging behavior (root category now uses rdk_console instead of rdk_syslog; console/syslog layouts changed). This is potentially breaking for consumers relying on previous defaults, and it isn’t mentioned in the PR description. Please confirm it’s intended and document the behavior change (or preserve existing defaults).

Copilot uses AI. Check for mistakes.
Comment on lines +83 to +85
FILE* file = fopen("cpu.txt", "a+");
if (file == NULL) {
perror("Error opening cpu.txt");
Copy link

Copilot AI Feb 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This tool writes CPU metrics to a relative path (cpu.txt) and exits on failure. If the program is run from a non-writable working directory (common on embedded/rootfs setups), it will fail immediately. Consider writing to a known-writable location like /tmp by default, or allow the output path to be configured via argv/env.

Suggested change
FILE* file = fopen("cpu.txt", "a+");
if (file == NULL) {
perror("Error opening cpu.txt");
const char *cpuPath = getenv("RDK_LOGGER_CPU_FILE");
if (cpuPath == NULL || cpuPath[0] == '\0') {
cpuPath = "/tmp/cpu.txt";
}
FILE* file = fopen(cpuPath, "a+");
if (file == NULL) {
fprintf(stderr, "Error opening CPU metrics file at '%s'\n", cpuPath);
perror("fopen");

Copilot uses AI. Check for mistakes.
Comment on lines +501 to +505
while (fgets(line, sizeof(line), logFile)) {
// Thread ID format typically appears as [TID:xxxxx] or similar
if (strstr(line, "TID") != NULL || strchr(line, '[') != NULL) {
foundThreadId = true;
break;
Copy link

Copilot AI Feb 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The thread-ID assertion is too permissive: strchr(line, '[') will be true for any log line because the level is always wrapped in brackets (e.g., [INFO ]). This means the test can pass even if the TID is not present. Please assert on a more specific pattern (e.g., a bracketed numeric token after the level, or a regex like \[[0-9]+\]).

Copilot uses AI. Check for mistakes.
Comment on lines +559 to +563
}
// Look for thread ID
if (strstr(line, "TID") != NULL || strchr(line, '[') != NULL) {
foundThreadId = true;
}
Copy link

Copilot AI Feb 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same issue as above: the thread-ID check will pass for any line that contains '[' (the log level), so this test may pass even if RDKLOG_FORMAT_WITH_TS_TID doesn't include the TID. Please assert specifically for the TID token (e.g., bracketed digits) rather than a generic '['/"TID" substring.

Copilot uses AI. Check for mistakes.
Copilot AI review requested due to automatic review settings February 22, 2026 05:39
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 9 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +556 to +559
// Look for timestamp patterns (date/time format)
if (strchr(line, ':') != NULL && strchr(line, '-') != NULL) {
foundTimestamp = true;
}
Copy link

Copilot AI Feb 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The timestamp validation is too broad. Checking for ':' and '-' characters could match unrelated content in the log message itself, not just the timestamp format. The actual timestamp format from line 742 in src/rdk_debug_priv.c is "%s.%06ld [%-5s] [%ld] %s\n" where the timestamp contains microseconds. Consider using a more specific pattern to validate the actual timestamp format.

Copilot uses AI. Check for mistakes.
struct stat st;
EXPECT_EQ(stat(logFilePath, &st), 0) << "Log file should exist";

// Read and verify log content contains both timestamp and thread ID
Copy link

Copilot AI Feb 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment states "Read and verify log content contains both timestamp and thread ID" but the test is for RDKLOG_FORMAT_DETAIL_WITHOUT_TS format which should not contain a timestamp according to line 587. The actual test only checks for module name. Either the comment is incorrect or the test logic is incomplete.

Suggested change
// Read and verify log content contains both timestamp and thread ID
// Read and verify log content contains the module name in detailed format (without timestamp)

Copilot uses AI. Check for mistakes.
testPolicy.fileLocation, testPolicy.fileName);
printf("logFilePath = %s\n", logFilePath);
struct stat st;
EXPECT_EQ(lstat(logFilePath, &st), 0) << "Log file should exist";
Copy link

Copilot AI Feb 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test uses lstat instead of stat, which is inconsistent with the other format tests (FormatWithTSTID at line 547 and FormatDetailWithoutTS at line 606) that use stat. Since the test is checking for a regular log file and not a symbolic link, stat should be used for consistency.

Suggested change
EXPECT_EQ(lstat(logFilePath, &st), 0) << "Log file should exist";
EXPECT_EQ(stat(logFilePath, &st), 0) << "Log file should exist";

Copilot uses AI. Check for mistakes.
char logFilePath[512];
snprintf(logFilePath, sizeof(logFilePath), "%s/%s",
testPolicy.fileLocation, testPolicy.fileName);
printf("logFilePath = %s\n", logFilePath);
Copy link

Copilot AI Feb 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The line uses tabs instead of spaces for indentation, which is inconsistent with the rest of the file that uses spaces. This creates visual inconsistency and may cause issues with code formatting tools.

Copilot uses AI. Check for mistakes.
Comment on lines +501 to +506
while (fgets(line, sizeof(line), logFile)) {
// Thread ID format typically appears as [TID:xxxxx] or similar
if (strstr(line, "TID") != NULL || strchr(line, '[') != NULL) {
foundThreadId = true;
break;
}
Copy link

Copilot AI Feb 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The validation logic is too broad and may produce false positives. The test searches for any line containing "TID" or '[' characters, which could match unrelated content. The actual format should be "[%ld]" for thread ID according to line 762 in src/rdk_debug_priv.c. Consider using a more specific pattern or regex to validate the actual thread ID format.

Copilot uses AI. Check for mistakes.
Comment on lines +560 to +563
// Look for thread ID
if (strstr(line, "TID") != NULL || strchr(line, '[') != NULL) {
foundThreadId = true;
}
Copy link

Copilot AI Feb 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The thread ID validation is too broad and may produce false positives. The test searches for any line containing "TID" or '[' characters, which could match unrelated content. The actual format should be "[%ld]" for thread ID according to line 746 in src/rdk_debug_priv.c. Consider using a more specific pattern or regex to validate the actual thread ID format.

Copilot uses AI. Check for mistakes.
fprintf(stderr, "Failed to parse CPU statistics from /proc/stat (parsed %d fields)\n", parsed);
fclose(file);
exit(EXIT_FAILURE);
}
Copy link

Copilot AI Feb 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If fgets returns NULL (e.g., due to an empty file or read error), the function exits normally but leaves the times structure zeroed out. This could lead to incorrect CPU usage calculations. Consider adding an error check or exit when fgets fails to read the first line.

Suggested change
}
}
} else {
if (feof(file)) {
fprintf(stderr, "Unexpected end of file while reading /proc/stat\n");
} else {
perror("Error reading /proc/stat");
}
fclose(file);
exit(EXIT_FAILURE);

Copilot uses AI. Check for mistakes.
Comment on lines +782 to +790
/* avoid redundant \n */
if ((n >= 2) &&
(n < event->evt_buffer.buf_size) &&
(event->evt_buffer.buf_data[n] == '\0') &&
(event->evt_buffer.buf_data[n - 1] == '\n') &&
(event->evt_buffer.buf_data[n - 2] == '\n'))
{
event->evt_buffer.buf_data[n - 1] = '\0';
}
Copy link

Copilot AI Feb 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The logic for detecting redundant newlines appears incorrect. Line 785 checks if buf_data[n] equals '\0', but n represents the number of characters written (excluding null terminator). The check should be for buf_data[n-1] and buf_data[n-2] only. Additionally, when n >= buf_size (truncation case), this check accesses beyond the valid buffer range.

Copilot uses AI. Check for mistakes.
rdk_Error ret = rdk_logger_ext_init(&config);
ASSERT_EQ(ret, RDK_SUCCESS) << "Extended initialization with RDKLOG_FORMAT_WITH_TID should succeed";

system("mkdir -p /tmp/rdk_logger_rotation_test");
Copy link

Copilot AI Feb 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The line uses tabs instead of spaces for indentation, which is inconsistent with the rest of the file that uses spaces. This creates visual inconsistency and may cause issues with code formatting tools.

Suggested change
system("mkdir -p /tmp/rdk_logger_rotation_test");
system("mkdir -p /tmp/rdk_logger_rotation_test");

Copilot uses AI. Check for mistakes.
Reason for change: Removed Log truncation & added testApp to measure CPU
Test Procedure: Verify huge data
Risks: Low
Co-authored-by: karuna2git <48997923+karuna2git@users.noreply.github.com>
Signed-off-by: Karunakaran A <karunakaran_amirthalingam@cable.comcast.com>
Copilot AI review requested due to automatic review settings February 22, 2026 05:51
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 6 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +49 to +54
if ((!format) && (!module))
{
va_start(args, format);
rdk_dbg_priv_log_msg(level, module, format, args);
va_end(args);
}
Copy link

Copilot AI Feb 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The null pointer check logic is inverted. The condition uses AND (&&) when it should use OR (||). Currently, the function only logs when BOTH format AND module are NULL, which is incorrect. It should log when EITHER format OR module is NOT NULL. Change the condition to if (format || module) to correctly allow logging when at least one of these parameters is valid.

Copilot uses AI. Check for mistakes.
Comment on lines +69 to +74
if ((!format) && (!module))
{
va_start(args, format);
rdk_dbg_priv_log_msg(level, module, format, args);
va_end(args);
}
Copy link

Copilot AI Feb 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The null pointer check logic is inverted. The condition uses AND (&&) when it should use OR (||). Currently, the function only logs when BOTH format AND module are NULL, which is incorrect. It should log when EITHER format OR module is NOT NULL. Change the condition to if (format || module) to correctly allow logging when at least one of these parameters is valid.

Copilot uses AI. Check for mistakes.
Comment on lines +79 to +80
if ((!format) && (!module))
rdk_dbg_priv_log_msg(level, module, format, args);
Copy link

Copilot AI Feb 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The null pointer check logic is inverted. The condition uses AND (&&) when it should use OR (||). Currently, the function only logs when BOTH format AND module are NULL, which is incorrect. It should log when EITHER format OR module is NOT NULL. Change the condition to if (format || module) to correctly allow logging when at least one of these parameters is valid.

Copilot uses AI. Check for mistakes.
Comment on lines +552 to +561
while (fgets(line, sizeof(line), logFile)) {
// Look for timestamp patterns (date/time format)
if (strchr(line, ':') != NULL && strchr(line, '-') != NULL) {
foundTimestamp = true;
}
// Look for thread ID
if (strstr(line, "TID") != NULL || strchr(line, '[') != NULL) {
foundThreadId = true;
}
if (foundTimestamp && foundThreadId) break;
Copy link

Copilot AI Feb 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The validation logic for detecting thread ID and timestamp is too broad. The condition strchr(line, '[') != NULL will match any line containing a square bracket, including the priority field (e.g., "[INFO]"), and the timestamp check strchr(line, ':') != NULL && strchr(line, '-') != NULL could match the colon in the microsecond timestamp or other unrelated content. This makes the test less reliable. Consider making the checks more specific by looking for actual thread ID patterns or timestamp formats.

Copilot uses AI. Check for mistakes.

<config>
<bufsize>1024</bufsize>
<bufsize>0</bufsize>
Copy link

Copilot AI Feb 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Setting bufsize to 0 in the log4c configuration means the buffer will dynamically grow as needed via realloc. However, there's a critical bug in the realloc implementation at src/rdk_debug_priv.c:778 where the return value is not checked for NULL. This configuration change makes the memory allocation failure path more likely to be hit with large log messages. The realloc bug must be fixed before this configuration change can be safely deployed.

Suggested change
<bufsize>0</bufsize>
<bufsize>2048</bufsize>

Copilot uses AI. Check for mistakes.
Comment on lines +85 to +86
if ((!format) && (!module))
rdk_dbg_priv_log_msg(level, module, format, args);
Copy link

Copilot AI Feb 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The null pointer check logic is inverted. The condition uses AND (&&) when it should use OR (||). Currently, the function only logs when BOTH format AND module are NULL, which is incorrect. It should log when EITHER format OR module is NOT NULL. Change the condition to if (format || module) to correctly allow logging when at least one of these parameters is valid.

Copilot uses AI. Check for mistakes.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings February 22, 2026 05:59
@karuna2git karuna2git closed this Feb 22, 2026
@github-actions github-actions bot locked and limited conversation to collaborators Feb 22, 2026
@karuna2git karuna2git deleted the topic/LogHugeData branch February 22, 2026 06:01
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants