Diff Report

	nvhealth-bugs.log			nvhealth-normal.log
f	1	Info	f	1	Info
	2	----		2	----
n	3	Timestamp: Tue Aug 14 14:15:43 2018 -0500	n	3	Timestamp: Tue Aug 14 14:28:15 2018 -0500
	4	Version: 18.04-4		4	Version: 18.04-4
	5			5
	6	Checks		6	Checks
	7	------		7	------
	8	DGX BaseOS Version [3.1.7]...........................................		8	DGX BaseOS Version [3.1.7]...........................................
	23	GPU link speed [0000:0e:00.0][8GT/s]................................. Healthy		23	GPU link speed [0000:0e:00.0][8GT/s]................................. Healthy
	24	GPU link width [0000:0e:00.0][x16]................................... Healthy		24	GPU link width [0000:0e:00.0][x16]................................... Healthy
	25	GPU link speed [0000:07:00.0][8GT/s]................................. Healthy		25	GPU link speed [0000:07:00.0][8GT/s]................................. Healthy
	26	GPU link width [0000:07:00.0][x16]................................... Healthy		26	GPU link width [0000:07:00.0][x16]................................... Healthy
	27	Verify PCIe switches................................................. Healthy		27	Verify PCIe switches................................................. Healthy
n	28	[sanity] Page retirement support on GPU None......................... Unknown	n
	29	Could not find key "output.nvidia_smi_log.gpu.retired_pages.double_b
	30	it_retirement.retired_count"
	31	[sanity] Return value of 'nvidia-smi nvlink -s' command.............. Unhealthy
	32	/usr/share/nvhealth/collect/nvidia-smi/nvidia-smi-nvlink-status.sh:
	33	line 14: [: too many arguments
	34	ERROR: This version of nvidia-smi does not query NVLink speed
	35	ERROR: NVLink speed query requires NVIDIA driver version 384.98 or
	36	later
	37
	38	nvidia-smi-nvlink-status.sh returned '1'
	39	Observed value "1" when "0" was expected
	40	[sanity] Parsing 'nvidia-smi' topology matrx......................... Unhealthy
	41	nvidia-smi-topo.py returned '1'
	42	Observed value "1" when "0" was expected
	43	NVIDIA Driver Version [None].........................................		28	NVIDIA Driver Version [384.145]......................................
	44	Inforom Storage Version [GPU None][None].............................		29	Inforom Storage Version [GPU 0][G500.0201.00.02].....................
				30	Retired pages pending [GPU 0][No].................................... Healthy
				31	NVLink Topology...................................................... Healthy
				32	Checking NVLink speed [GPU 0 Link 0][25.781 GB/s].................... Healthy
				33	Checking NVLink speed [GPU 0 Link 1][25.781 GB/s].................... Healthy
				34	Checking NVLink speed [GPU 0 Link 2][25.781 GB/s].................... Healthy
				35	Checking NVLink speed [GPU 0 Link 3][25.781 GB/s].................... Healthy
				36	Checking NVLink speed [GPU 1 Link 0][25.781 GB/s].................... Healthy
				37	Checking NVLink speed [GPU 1 Link 1][25.781 GB/s].................... Healthy
				38	Checking NVLink speed [GPU 1 Link 2][25.781 GB/s].................... Healthy
				39	Checking NVLink speed [GPU 1 Link 3][25.781 GB/s].................... Healthy
				40	Checking NVLink speed [GPU 2 Link 0][25.781 GB/s].................... Healthy
				41	Checking NVLink speed [GPU 2 Link 1][25.781 GB/s].................... Healthy
				42	Checking NVLink speed [GPU 2 Link 2][25.781 GB/s].................... Healthy
				43	Checking NVLink speed [GPU 2 Link 3][25.781 GB/s].................... Healthy
				44	Checking NVLink speed [GPU 3 Link 0][25.781 GB/s].................... Healthy
				45	Checking NVLink speed [GPU 3 Link 1][25.781 GB/s].................... Healthy
				46	Checking NVLink speed [GPU 3 Link 2][25.781 GB/s].................... Healthy
				47	Checking NVLink speed [GPU 3 Link 3][25.781 GB/s].................... Healthy
				48	Total retired page count [GPU 0][0 retired pages].................... Healthy
	45	VBIOS Version [GPU None][None].......................................		49	VBIOS Version [GPU 0][88.00.24.00.01]................................
	46	Inforom Storage Version [GPU 1][G500.0201.00.02].....................		50	Inforom Storage Version [GPU 1][G500.0201.00.02].....................
n	47	Retired pages pending [GPU None][None]............................... Unknown	n
	48	Could not find key
	49	"output.nvidia_smi_log.gpu.retired_pages.pending_retirement"
	50	Suggested action: Reboot the system to blacklist GPU pages flagged
	51	for retirement. See: http://docs.nvidia.com/deploy/dynamic-page-
	52	retirement/index.html#blacklisting
	53	Total retired page count [GPU None][None retired pages].............. Unknown
	54	Could not find key
	55	"output.nvidia_smi_log.gpu.retired_pages.total_retired_count"
	56	Retired pages pending [GPU 1][No].................................... Healthy		51	Retired pages pending [GPU 1][No].................................... Healthy
	57	Total retired page count [GPU 1][0 retired pages].................... Healthy		52	Total retired page count [GPU 1][0 retired pages].................... Healthy
	58	VBIOS Version [GPU 1][88.00.24.00.01]................................		53	VBIOS Version [GPU 1][88.00.24.00.01]................................
	59	Inforom Storage Version [GPU 2][G500.0201.00.02].....................		54	Inforom Storage Version [GPU 2][G500.0201.00.02].....................
	60	Retired pages pending [GPU 2][No].................................... Healthy		55	Retired pages pending [GPU 2][No].................................... Healthy
	62	VBIOS Version [GPU 2][88.00.24.00.01]................................		57	VBIOS Version [GPU 2][88.00.24.00.01]................................
	63	Inforom Storage Version [GPU 3][G500.0201.00.02].....................		58	Inforom Storage Version [GPU 3][G500.0201.00.02].....................
	64	Retired pages pending [GPU 3][No].................................... Healthy		59	Retired pages pending [GPU 3][No].................................... Healthy
	65	Total retired page count [GPU 3][0 retired pages].................... Healthy		60	Total retired page count [GPU 3][0 retired pages].................... Healthy
	66	VBIOS Version [GPU 3][88.00.24.00.01]................................		61	VBIOS Version [GPU 3][88.00.24.00.01]................................
n	67	NVIDIA Driver Version [None].........................................	n
	68	Inforom Storage Version [GPU None][None].............................
	69	[sanity] Page retirement support on GPU None......................... Unknown
	70	Could not find key "output.nvidia_smi_log.gpu.retired_pages.double_b
	71	it_retirement.retired_count"
	72	Retired pages pending [GPU None][None]............................... Unknown
	73	Could not find key
	74	"output.nvidia_smi_log.gpu.retired_pages.pending_retirement"
	75	Suggested action: Reboot the system to blacklist GPU pages flagged
	76	for retirement. See: http://docs.nvidia.com/deploy/dynamic-page-
	77	retirement/index.html#blacklisting
	78	Total retired page count [GPU None][None retired pages].............. Unknown
	79	Could not find key
	80	"output.nvidia_smi_log.gpu.retired_pages.total_retired_count"
	81	VBIOS Version [GPU None][None].......................................
	82	NVIDIA Driver Version [None].........................................
	83	Inforom Storage Version [GPU None][None].............................
	84	[sanity] Page retirement support on GPU None......................... Unknown
	85	Could not find key "output.nvidia_smi_log.gpu.retired_pages.double_b
	86	it_retirement.retired_count"
	87	Retired pages pending [GPU None][None]............................... Unknown
	88	Could not find key
	89	"output.nvidia_smi_log.gpu.retired_pages.pending_retirement"
	90	Suggested action: Reboot the system to blacklist GPU pages flagged
	91	for retirement. See: http://docs.nvidia.com/deploy/dynamic-page-
	92	retirement/index.html#blacklisting
	93	Total retired page count [GPU None][None retired pages].............. Unknown
	94	Could not find key
	95	"output.nvidia_smi_log.gpu.retired_pages.total_retired_count"
	96	VBIOS Version [GPU None][None].......................................
	97	Verify installed disks............................................... Healthy		62	Verify installed disks............................................... Healthy
	98	Linux kernel version [4.4.0-127-generic].............................		63	Linux kernel version [4.4.0-127-generic].............................
n	99	System Uptime [up 3 days, 20 hours, 20 minutes ].....................	n	64	System Uptime [up 5 minutes ]........................................
	100	Verify DIMM vendors.................................................. Healthy		65	Verify DIMM vendors.................................................. Healthy
	101			66
	102	System Summary		67	System Summary
	103	--------------		68	--------------
	104	Product Name: DGX Station		69	Product Name: DGX Station
	105	Manufacturer: NVIDIA		70	Manufacturer: NVIDIA
	106	DGX Serial Number: 0154217000027		71	DGX Serial Number: 0154217000027
n	107	Uptime: up 3 days, 20 hours, 20 minutes	n	72	Uptime: up 5 minutes
	108	Motherboard:		73	Motherboard:
	109	BIOS Version: 0406		74	BIOS Version: 0406
	110	Serial Number: 170295316900143		75	Serial Number: 170295316900143
	111	GPU:		76	GPU:
n	112	NVIDIA Driver Version: Unknown	n	77	NVIDIA Driver Version: 384.145
	113	Product Name(s): Unknown		78	Product Name(s): Tesla V100-DGXS-16GB
	114	VBIOS Version(s): Unknown		79	VBIOS Version(s): 88.00.24.00.01
	115	Software:		80	Software:
	116	DGX BaseOS Version: 3.1.7		81	DGX BaseOS Version: 3.1.7
	117	Kernel Version: 4.4.0-127-generic		82	Kernel Version: 4.4.0-127-generic
	118			83
	119	Health Summary		84	Health Summary
	120	--------------		85	--------------
n	121	25 out of 36 checks are Healthy	n	86	44 out of 44 checks are Healthy
	122	2 out of 36 checks are Unhealthy
	123	9 out of 36 checks are Unknown
	124	Overall system status is Unhealthy		87	Overall system status is Healthy
	125			88
t	126	Problem detected.	t
	127	Please visit the ESP portal: https://nvid.nvidia.com/enterpriselogin
	128	And create a support ticket with the log file attached.
	129
	130	Log file written to: /tmp/nvhealth-log.kE3XdxPBlo.jsonl		89	Log file written to: /tmp/nvhealth-log.q4DASkL1aD.jsonl