Netstat #70

sreedharamzn · 2023-07-24T13:23:43Z

Collect and visualize netstat data. aperf was used to collect data during a network load simulation exercise.

aperf_report_netstat_simulation.tar.gz

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

janaknat · 2023-07-24T21:06:36Z

src/data/netstat.rs

+    fn collect_data(&mut self) -> Result<()> {
+        self.time = TimeEnum::DateTime(Utc::now());
+        self.data = String::new();
+        self.data = std::fs::read_to_string("/proc/net/netstat")?;


What is the average increase in time taken for aperf before and after this change?

100 samples collected on Ubuntu 22.04 (m6a.4xlarge) ec2 instance:
Average Time (with Netstat) = 4.316205832ms
Average Time (without Netstat) = 4.198065059ms
Average increase in Time = 0.118140773ms

100 samples collected on AL Cloud Desktop machine:
Average Time (with Netstat) = 10.06244311ms
Average Time (without Netstat) = 9.958092307ms
Average increase in Time = 0.104350803ms

That's a big difference in times between the two environments (I mean regardless of your change). The cloud dev desktop is sporting a higher CPU count, right?

My cloud dev desktop is having same CPU count (16) as that of m6a.4xlarge ec2 instance.

However, sizes of CPU cache & main memory of my cloud dev desktop, is significantly lesser than that of m6a.4xlarge ec2 instance. (documented it below for your reference)

Cloud dev desktop
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
CPU(s): 16
Model name: Intel(R) Xeon(R) CPU E5@ 2.90GHz
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 25600K
MemTotal: 30779248 kB

m6a.4xlarge ec2 instance
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
CPU(s): 16
Model name: AMD EPYC 7R13 Processor
L1d cache: 256 KiB (8 instances
L1i cache: 256 KiB (8 instances)
L2 cache: 4 MiB (8 instances)
L3 cache: 32 MiB (1 instance)
MemTotal: 64191680 kB

Most likely aperf isn't (for one or two reasons) collecting PMU stats on your AMD instance, resulting in lower collection times.

You are correct. PMU stats collection is getting skipped because of unsupported CPU.

"[2023-07-26T20:15:47Z ERROR aperf_lib] Excluding perf_stat from collection. Error msg: Unsupported CPU"

janaknat · 2023-07-24T21:08:30Z

src/data/netstat.rs

+        let mut lines = reader.lines();
+
+        while let (
+            Some(line1), Some(line2)) = (lines.next(), lines.next()) {


What happens if line1 has a value and line2 is None. Will it continue? It might cause issues later on since we blindly use .unwrap().

As per '/proc/net/netstat' data contract, header row will always succeeded by a data row.
So, line2 is always expected to be present when line1 is available.

I also wanted the tool to fail, if the data contract is violated so we get noticed and can take appropriate action.
But let me know, if this needs to be absorbed and no exception should be raised in the unlikely scenario when line2 is None.
I'll take care of the necessary changes.

janaknat · 2023-07-24T21:10:15Z

src/data/netstat.rs

+
+            for name in names {
+                let val = values.next().ok_or(PDError::ProcessorOptionExtractError)?;
+                map.insert(tag.to_owned() + " " + &name.to_owned(), val.parse::<i64>()?);


Do we ever expect the value to be signed?

Not really. Have changed this to u64

janaknat · 2023-07-24T21:12:31Z

src/data/netstat.rs

+}
+
+fn get_entries(value: Netstat) -> Result<String> {
+    let mut keys: Vec<String> = value.netstat_data.into_keys().collect();


A HashMap does not maintain the order of the keys. Every hashmap will produce a different output in the report.

I agree. Currently i have a sorting logic in place for ordering the plots lexicographically. In my next CR, i'll change this to a Vector and get rid of the sorting logic.

janaknat · 2023-07-24T21:12:54Z

src/data/netstat.rs

+
+fn get_entries(value: Netstat) -> Result<String> {
+    let mut keys: Vec<String> = value.netstat_data.into_keys().collect();
+    keys.sort();


Why do we sort the keys?

For ordering the plots lexicographically based on the parameter names.

janaknat · 2023-07-24T21:14:00Z

src/data/netstat.rs

+                ProcessedData::Netstat(ref value) => values.push(value.clone()),
+                _ => unreachable!(),
+            }
+        }


There should be a check for the number of params here and in the "values" call.

Added the check and panicking if params & values count differ.

janaknat · 2023-07-24T21:15:17Z

src/data/netstat.rs

+        assert!(netstat.collect_data().unwrap() == ());
+        buffer.push(Data::NetstatRaw(netstat));
+        processed_buffer.push(Netstat::new().process_raw_data(buffer[0].clone()).unwrap());
+        let json = Netstat::new().get_data(processed_buffer, "run=test&get=values&key=IpExt: InNoECTPkts".to_string()).unwrap();


Is this guaranteed to be there in the build system where the unit test will be run?

I have tested this on AL 2, Ubuntu 22.04 & RHEL 9 ec2 instances and the unit test always succeeded.

janaknat · 2023-07-24T21:19:07Z

src/html_files/netstat.ts

+            title: 'Time (s)',
+        },
+        yaxis: {
+            title: 'Packets',


Is this true for all the keys?

Have changed this to 'Count', to keep it generic. Will work on a separate CR to add custom Units.

janaknat · 2023-07-24T21:20:20Z

src/data/netstat.rs

+            .ok_or(PDError::VisualizerNetstatValueGetError(key.to_string()))?;
+
+        let mut v = *curr_value;
+        if !key.contains("nr_") {


Why is this check here?

wash-amzn · 2023-07-26T18:25:23Z

src/data/netstat.rs

+    fn test_collect_data() {
+        let mut netstat = NetstatRaw::new();
+
+        assert!(netstat.collect_data().unwrap() == ());


Don't change it if it's what all the other tests in the project are doing, but assert!(result.is_ok()) would be nicer than assert!(result.unwrap() == ())

since all other tests are using the same, left it unchanged.

wash-amzn · 2023-07-26T19:06:41Z

src/html_files/index.html

@@ -19,6 +19,7 @@ <h2>APerf</h2>
 			<button class="tablinks" name="interrupts">Interrupt Data</button>
 			<button class="tablinks" name="disk_stats">Disk Stats</button>
 			<button class="tablinks" name="perfstat">PMU Stats</button>
+			<button class="tablinks" name="netstat">NET Stat</button>


NET isn't an acronym ;)

Ok. Changed the name to 'Netstat'

To be consistent with the others, I'd say it should be "Net Stats"

wash-amzn

Janak also has to approve since he had review feedback

sreedharamzn added 3 commits July 24, 2023 07:34

Add support for collecting/visualizing Netstat data

15819bd

removed commented code

1224571

renamed x-axis label to 'Packets'

99d0cf7

sreedharamzn requested review from janaknat and wash-amzn July 24, 2023 13:23

janaknat reviewed Jul 24, 2023

View reviewed changes

addressed feedback

1887025

wash-amzn reviewed Jul 26, 2023

View reviewed changes

sreedharamzn added 2 commits July 26, 2023 20:18

made changes to display the error message

604bad3

renamed button name to 'Net Stats' as per the feedback

6e27799

wash-amzn approved these changes Jul 26, 2023

View reviewed changes

janaknat approved these changes Jul 26, 2023

View reviewed changes

janaknat merged commit 9905a3e into main Jul 26, 2023

janaknat deleted the netstat branch August 15, 2023 16:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Netstat #70

Netstat #70

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Netstat #70

Netstat #70

Uh oh!

Conversation

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!