• Unable to achieve gigabit speeds
    29 replies, posted
I just put together an Ubuntu server for basic file storage / sharing, but when attempting to move data between the two machines I can only achieve a 15MB/s data transfer rate. I've got my personal Windows machine and server connected to a TL-SG1008D gigabit switch via Cat6 cables, and both onboard ports are rated for 10/100/1000. The Windows machine uses a Sandisk Ultra II and the server uses two 1TB Western Digital Blacks running in RAID 0 (just playing around right now so I'm not worried about RAID failure), and they're brand new. I've run an IPERF test between the two machines and the end result was 1.10 GBytes 942 Mbits/sec - so this might be a Windows / storage media issue. [t]http://i.imgur.com/M9Ccyme.png[/t] Ran hdparm for both drives in the server - everything seems to check out. /dev/sda: Timing cached reads: 9400 MB in 2.00 seconds = 4702.67 MB/sec Timing buffered disk reads: 550 MB in 3.01 seconds = 182.83 MB/sec james@Moonunit:~$ sudo hdparm -tT /dev/sdb /dev/sdb: Timing cached reads: 9530 MB in 2.00 seconds = 4768.41 MB/sec Timing buffered disk reads: 538 MB in 3.01 seconds = 178.82 MB/sec I'm unsure where the bottleneck is - any ideas?
Turn off Windows firewall?
[QUOTE=MTMod;50607877]Turn off Windows firewall?[/QUOTE] No dice :c
Disable Large send offload and reboot You can do this on the advanced tab on your NIC from the device manager Its suppose to reduce cpu overhead such and such but its been problematic since 2001
[QUOTE=Saxon;50608006]Disable Large send offload and reboot You can do this on the advanced tab on your NIC from the device manager Its suppose to reduce cpu overhead such and such but its never worked properly since its inception in 2001[/QUOTE] No luck, even disabled flow control. Could it be an issue with my switch despite the fact that IPERF shows full gigabit speeds? Drives are brand new, both NICs are fine, cat6 rj-45s, and the switch are the variables...
Does your switch support jumbo frames? Some people report better speeds by enabling that option, I use it at home to stream HD video to smart tv's from my own PC I use an old SRW2024 though and it has to be toggled, not sure what yours is like
What's the server hardware, CPU / RAM wise?
[QUOTE=Cold;50608092]What's the server hardware, CPU / RAM wise?[/QUOTE] Server CPU: AMD FX-6300 (From an old machine) and 16gb ram. [editline]27th June 2016[/editline] [QUOTE=Saxon;50608064]Does your switch support jumbo frames? Some people report better speeds by enabling that option, I use it at home to stream HD video to smart tv's from my own PC I use an old SRW2024 though and it has to be toggled, not sure what yours is like[/QUOTE] The switch is a cheap TP-LINK TL-SG1008D, but the specs seemed up to snuff, though at this point I'm starting to suspect it's the source of this issue. It was a steal at $20, but you get what you pay for I suppose.
Unlikely the switch is the cause. I'm not exactly sure where I can reference you - Linus Tech Tips did cover him having similar issues, but a lot of tweaking resolved it. How exactly are you transferring? What software are you using on your server for access?
[QUOTE=Killervalon;50611695]Unlikely the switch is the cause. I'm not exactly sure where I can reference you - Linus Tech Tips did cover him having similar issues, but a lot of tweaking resolved it. How exactly are you transferring? What software are you using on your server for access?[/QUOTE] I'm using Samba as my file share. Simply copying and pasting does the trick, though using an SFTP protocol yields the same results.
replace your cables or switch them around. I've had them fail every now and then and throw up a few bizarre situations. Even if they're new/tested its still worth a try before you go screwing around with tons of other stuff.
Check the performance of both machines while the transfer is running. I ran into a similar situation recently with 10G where we assumed the server hardware could handle it but it actually couldn't.
[QUOTE=MTMod;50616734]Check the performance of both machines while the transfer is running. I ran into a similar situation recently with 10G where we assumed the server hardware could handle it but it actually couldn't.[/QUOTE] [QUOTE=icarusfoundyou;50616594]replace your cables or switch them around. I've had them fail every now and then and throw up a few bizarre situations. Even if they're new/tested its still worth a try before you go screwing around with tons of other stuff.[/QUOTE] Never thought of the cables or server load being the issue - I'll definitely check it out when I get the chance. [editline]29th June 2016[/editline] [QUOTE=MTMod;50616734]Check the performance of both machines while the transfer is running. I ran into a similar situation recently with 10G where we assumed the server hardware could handle it but it actually couldn't.[/QUOTE] CPU didn't exceed 5% of it's usage through a simple file transfer and IPERF test - the only difference between the two was 248Mbps vs 956Mpbs for the transfer and IPERF respectively. Though there was one little nugget of information I completely forgot about... there's only 4gb of ram in the system. I'll throw in two more sticks when I get back, and hopefully this resolves the issue... feel like a big ol' dummy.
Update: Throwing in a full 16gb RAM and switching the CAT6 cables did nothing :c During a file transfer CPU usage still doesn't go past 4%.
What exactly is your network topology? You said your switch is 1000 and so are your end devices, but is there anything else in between they hit? IE a router or physical firewall?
[QUOTE=ghostofme;50619773]What exactly is your network topology? You said your switch is 1000 and so are your end devices, but is there anything else in between they hit? IE a router or physical firewall?[/QUOTE] They're both connected to the switch, which is itself connected to the router downstairs via a 50ft cat6. My roommate's PC is also on the same switch, and I'm able to get a 65-75 MBps transfer speed with strange dips to 45MBps. The same transfer to my server only yields 15MBps, but it's much steadier than the Windows to Windows transfer. While it's still not full gigabit, it's a step in the right direction... I wonder what could be going on in Ubuntu that's stifling my transfer speed. Behold my formidable paint skills: [t]http://i.imgur.com/8akYLxx.jpg[/t]
Could you post your /etc/samba/smb.conf file? [editline]29th June 2016[/editline] And I noticed you have raw disc read speeds, but not off the file system. Try using 'pv' on a large file to see what read speeds you get from the raid array: [code]pv largefile > /dev/null[/code] make sure you reboot or drop the caches before so you aren't just reading data cached in ram [code]echo 3 > /proc/sys/vm/drop_caches[/code]
[QUOTE=IpHa;50619936]Could you post your /etc/samba/smb.conf file? [editline]29th June 2016[/editline] And I noticed you have raw disc read speeds, but not off the file system. Try using 'pv' on a large file to see what read speeds you get from the raid array: [code]pv largefile > /dev/null[/code] make sure you reboot or drop the caches before so you aren't just reading data cached in ram [code]echo 3 > /proc/sys/vm/drop_caches[/code][/QUOTE] Man, the transfer speed fluctuates constantly, capping at around 370MB/s and going as low as the KB range - weird... How can this be with a set of brand new drives? [code][global] ; General server settings netbios name = moonunit server string = workgroup = nexus announce version = 5.0 socket options = TCP_NODELAY IPTOS_LOWDELAY SO_KEEPALIVE SO_RCVBUF=8192 SO_SNDBUF=8192 passdb backend = tdbsam security = user null passwords = true username map = /etc/samba/smbusers name resolve order = hosts wins bcast wins support = yes printing = CUPS printcap name = CUPS syslog = 1 syslog only = yes ; NOTE: If you need access to the user home directories uncomment the ; lines below and adjust the settings to your hearts content. ;[homes] ;valid users = %S ;create mode = 0600 ;directory mode = 0755 ;browseable = no ;read only = no ;veto files = /*.{*}/.*/mail/bin/ ; NOTE: Only needed if you run samba as a primary domain controller. ; Not needed as this config doesn't cover that matter. ;[netlogon] ;path = /var/lib/samba/netlogon ;admin users = Administrator ;valid users = %U ;read only = no ; NOTE: Again - only needed if you're running a primary domain controller. ;[Profiles] ;path = /var/lib/samba/profiles ;valid users = %U ;create mode = 0600 ;directory mode = 0700 ;writeable = yes ;browseable = no ; NOTE: Inside this place you may build a printer driver repository for ; Windows - I'll cover this topic in another HOWTO. [print$] path = /var/lib/samba/printers browseable = yes guest ok = yes read only = yes write list = root create mask = 0664 directory mask = 0775 [printers] path = /tmp printable = yes guest ok = yes browseable = no ; Uncomment if you need to share your CD-/DVD-ROM Drive ;[DVD-ROM Drive] ;path = /media/cdrom ;browseable = yes ;read only = yes ;guest ok = yes [files] path = /srv/samba/files browseable = yes read only = no guest ok = no valid users = @moonmen [test] path = /srv/samba/test browseable = yes read only = no guest ok = no create mask = 0644 directory mask = 0755 force group = @startest [james] path = /home/james/files valid users = james writeable = yes guest ok = no [/code]
Config file looks good, I'm thinking you have a bad drive or something wrong with the RAID array. Check the SMART data for each drive paying attention to the reallocated sector count and pending sector count. Anything other than 0 here is bad. [code]sudo smartctl -a /dev/sd*[/code] If that comes up good and you don't have any important data on the RAID array you could try formatting them as single drives and see if the performance problems follow a certain drive.
Have you tested out your cables? Might be faulty. It's' either that or one of the drives isn't running at full speed.
Have you tried directly connecting the machines with a crossover cable to see what happens?
[QUOTE=rhx123;50622705]Have you tried directly connecting the machines with a crossover cable to see what happens?[/QUOTE] This, but I'd like to point out that with modern gear - including [I]all[/I] Gigabit (as in, 1000BASE-T Ethernet) gear - distinguishing between crossover and patch cables is no longer required.
[QUOTE=IpHa;50622378]Config file looks good, I'm thinking you have a bad drive or something wrong with the RAID array. Check the SMART data for each drive paying attention to the reallocated sector count and pending sector count. Anything other than 0 here is bad. [code]sudo smartctl -a /dev/sd*[/code] If that comes up good and you don't have any important data on the RAID array you could try formatting them as single drives and see if the performance problems follow a certain drive.[/QUOTE] Looks like smartctl wasn't installed, and it comes up through apt as "...not available, but referred to by another package". Is there a comparable tool out there? [QUOTE=QuinnithXD;50622451]Have you tested out your cables? Might be faulty. It's' either that or one of the drives isn't running at full speed.[/QUOTE] Tried a few spare cables yesterday, none changed the outcome :c [QUOTE=rhx123;50622705]Have you tried directly connecting the machines with a crossover cable to see what happens?[/QUOTE] Took a little fanangling, but only a few MB difference overall.
[QUOTE=Falstad007;50624756]Looks like smartctl wasn't installed, and it comes up through apt as "...not available, but referred to by another package". Is there a comparable tool out there?[/QUOTE] The Ubuntu package containing the smartctl program is called "smartmontools".
[QUOTE=IpHa;50622378]Config file looks good, I'm thinking you have a bad drive or something wrong with the RAID array. Check the SMART data for each drive paying attention to the reallocated sector count and pending sector count. Anything other than 0 here is bad. [code]sudo smartctl -a /dev/sd*[/code] If that comes up good and you don't have any important data on the RAID array you could try formatting them as single drives and see if the performance problems follow a certain drive.[/QUOTE] [QUOTE=DrTaxi;50624891]The Ubuntu package containing the smartctl program is called "smartmontools".[/QUOTE] Derp, thanks a bunch! [code]james@Moonunit:~$ sudo smartctl -a /dev/sda smartctl 6.5 2016-01-24 r4214 [x86_64-linux-4.4.0-28-generic] (local build) Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Western Digital Black Device Model: WDC WD1003FZEX-00MK2A0 Serial Number: WD-WCC3F2JA3U75 LU WWN Device Id: 5 0014ee 2b7939197 Firmware Version: 01.01A01 User Capacity: 1,000,204,886,016 bytes [1.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Thu Jun 30 21:02:27 2016 EDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (11220) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off supp ort. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 116) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x3035) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_ FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0027 175 175 021 Pre-fail Always - 2208 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 12 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 624 10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 12 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 1 193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 10 194 Temperature_Celsius 0x0022 110 103 000 Old_age Always - 33 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 No self-tests have been logged. [To run self-tests, use: smartctl -t] SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. james@Moonunit:~$ sudo smartctl -a /dev/sdb smartctl 6.5 2016-01-24 r4214 [x86_64-linux-4.4.0-28-generic] (local build) Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Toshiba 3.5" DT01ACA... Desktop HDD Device Model: TOSHIBA DT01ACA100 Serial Number: Y2TVT2RPS LU WWN Device Id: 5 000039 ff2cc2c12 Firmware Version: MS2OA750 User Capacity: 1,000,204,886,016 bytes [1.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Thu Jun 30 21:02:43 2016 EDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x84) Offline data collection activity was suspended by an interrupting command from host. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 7653) seconds. Offline data collection capabilities: (0x5b) SMART execute Offline immediate. Auto Offline data collection on/off supp ort. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. No Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 128) minutes. SCT capabilities: (0x003d) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_ FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000b 100 100 016 Pre-fail Always - 0 2 Throughput_Performance 0x0005 141 141 054 Pre-fail Offline - 73 3 Spin_Up_Time 0x0007 127 127 024 Pre-fail Always - 182 (Average 180) 4 Start_Stop_Count 0x0012 100 100 000 Old_age Always - 2950 5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0 7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always - 0 8 Seek_Time_Performance 0x0005 113 113 020 Pre-fail Offline - 35 9 Power_On_Hours 0x0012 098 098 000 Old_age Always - 15100 10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 2891 192 Power-Off_Retract_Count 0x0032 098 098 000 Old_age Always - 2954 193 Load_Cycle_Count 0x0012 098 098 000 Old_age Always - 2954 194 Temperature_Celsius 0x0002 176 176 000 Old_age Always - 34 (Min/Max 9/53) 196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0 197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 1 SMART Error Log Version: 1 ATA Error Count: 1 CR = Command Register [HEX] FR = Features Register [HEX] SC = Sector Count Register [HEX] SN = Sector Number Register [HEX] CL = Cylinder Low Register [HEX] CH = Cylinder High Register [HEX] DH = Device/Head Register [HEX] DC = Device Command Register [HEX] ER = Error register [HEX] ST = Status register [HEX] Powered_Up_Time is measured from power on, and printed as DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes, SS=sec, and sss=millisec. It "wraps" after 49.710 days. Error 1 occurred at disk power-on lifetime: 7581 hours (315 days + 21 hours) When the command that caused the error occurred, the device was active or idle . After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 84 51 e0 60 e6 f4 06 Error: ICRC, ABRT at LBA = 0x06f4e660 = 116713056 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 61 20 e0 5f e6 f4 46 ff 09:08:21.989 WRITE FPDMA QUEUED 61 20 28 40 ec f4 40 00 09:08:21.987 WRITE FPDMA QUEUED 61 00 20 40 eb f4 40 00 09:08:21.987 WRITE FPDMA QUEUED 61 00 18 40 ea f4 40 00 09:08:21.987 WRITE FPDMA QUEUED 61 00 10 40 e9 f4 40 00 09:08:21.987 WRITE FPDMA QUEUED SMART Self-test log structure revision number 1 No self-tests have been logged. [To run self-tests, use: smartctl -t] SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. [/code] All looks clear :c
I don't see a problem either, but what I do see is that these aren't brand new drives. Anyway, have you tried protocols other than SMB and SFTP (which can be rather slow)? I'd try FTP and/or HTTP.
Do you have more than 2 hard drives? sdb is a TOSHIBA DT01ACA100 with 15100 power on hours, but you said you had 2x WD Blacks.
[QUOTE=IpHa;50625607]Do you have more than 2 hard drives? sdb is a TOSHIBA DT01ACA100 with 15100 power on hours, but you said you had 2x WD Blacks.[/QUOTE] I have no idea how I kept that old Toshiba in there... sorry guys :\ I'll break the array and rebuild with the solo WD Black see if that helps, though it was a problem when I was running with my previous drives (512gb Samsung HD502HJ). [editline]30th June 2016[/editline] [QUOTE=IpHa;50625607]Do you have more than 2 hard drives? sdb is a TOSHIBA DT01ACA100 with 15100 power on hours, but you said you had 2x WD Blacks.[/QUOTE] Either way, it's just the two drives - backing up and reverting to one now. [editline]30th June 2016[/editline] [QUOTE=DrTaxi;50625605]I don't see a problem either, but what I do see is that these aren't brand new drives. Anyway, have you tried protocols other than SMB and SFTP (which can be rather slow)? I'd try FTP and/or HTTP.[/QUOTE] Got the Blacks about two weeks ago, which in the grand scheme of HDD lifespans is brand new. [B] Update: [/B]After breaking the array and re-establishing Ubuntu server, I'm now averaging a 15MB/s rate upload and a 30MB/s with the single WD. I'm still able to get 80 - 90MB/s when copying a 5gb folder to my roommate's Window's machine, so it's definitely looking like an issue within Ubuntu, but I can't figure out what the bottleneck is.
Thanks to you guys I narrowed the problem down to my server, specifically Samba. I've solved the issue by removing the SO_SNDBUF and SO_RCVBUF=8192 from socket options. [t]http://i.imgur.com/7w1iRmD.png[/t] These two settings alone were gimping it's u/d speed by 100mbps. Thanks for all the assistance!
-ninja'd-
Sorry, you need to Log In to post a reply to this thread.