RAID Information and Storage information on the OLED

Playing with my new unit and I’ve setup a RAID1 with two 4TB disks as /dev/md0, I formatted the RAID and mounted it. The RAID Information and Storage Utilization pages on the OLED are showing that the RAID volume is 100% full but that is not the case. df shows that there is only 28K of the 3.4T of space? The Storage Utilization page shows what I expect for the root partition, i.e. the value reported on the OLED matches what df says the utilization is.

Is this a bug or am I misunderstanding what this is supposed to be showing?

@skymoo would you mind share more about your setup? Are you running OMW ? Would be mind share some screen cap on your findings vs the oled display ?

We think it might be a bug in our current script, but we would need more information to better diagnose it. Thank you.

I’m not using OMV, I just a lite Raspbian Bullseye installation and a manually created RAID (as OMV doesn’t currently support creating RAIDs using USB attached disks I couldn’t see the point of using it). Attached are a couple of pictures showing the RAID and Storage pages. Details of the RAID are:

pi@mysql:~ $ sudo mdadm --detail /dev/md0
/dev/md0:
           Version : 1.2
     Creation Time : Thu Feb  3 22:54:53 2022
        Raid Level : raid1
        Array Size : 3906885440 (3725.90 GiB 4000.65 GB)
     Used Dev Size : 3906885440 (3725.90 GiB 4000.65 GB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Fri Feb  4 09:01:17 2022
             State : active, resyncing 
    Active Devices : 2
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 0

Consistency Policy : bitmap

     Resync Status : 60% complete

              Name : mysql:0  (local to host mysql)
              UUID : 315249cd:d6a108ca:7eaadd93:00e9f251
            Events : 61913

    Number   Major   Minor   RaidDevice State
       0       8       17        0      active sync   /dev/sdb1
       1       8       33        1      active sync   /dev/sdc1
pi@mysql:~ $ 

and df reports the utilization as:

pi@mysql:~ $ df -h / /dev/md0
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       118G  2.0G  111G   2% /
/dev/md0        3.6T   28K  3.4T   1% /mnt
pi@mysql:~ $

img_0172
img_0173

I’m using OMV 6 for a RAID1 setup (configured via mdadm) and my display also shows the RAID volumes at 100% usage.

OMV only shows 52KB used.

My OLED information screens are showing similar information to what skymoo is seeing, except mine is 2TB instead of 4TB.

root@ArgonEON:~# mdadm --detail /dev/md127
/dev/md127:
Version : 1.2
Creation Time : Sun Jan 30 17:06:43 2022
Raid Level : raid1
Array Size : 1953382464 (1862.89 GiB 2000.26 GB)
Used Dev Size : 1953382464 (1862.89 GiB 2000.26 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent

 Intent Bitmap : Internal

   Update Time : Fri Feb  4 11:46:38 2022
         State : clean
Active Devices : 2

Working Devices : 2
Failed Devices : 0
Spare Devices : 0

Consistency Policy : bitmap

          Name : ArgonEON:vol1  (local to host ArgonEON)
          UUID : 74036276:72c842f0:c36b738b:9173893e
        Events : 3052

Number   Major   Minor   RaidDevice State
   0       8        0        0      active sync   /dev/sda
   1       8       32        1      active sync   /dev/sdc

df utilization reporting:

pi@ArgonEON:~ $ df -h / /dev/md127
Filesystem Size Used Avail Use% Mounted on
/dev/root 459G 1.9G 438G 1% /
/dev/md127 1.8T 52K 1.8T 1% /srv/dev-disk-by-uuid-dbaa8077-5198-4748-8cc1-3867ed4ea4df

Could it be as simple as … 100% of the RAW space is being utilized by RAID? Given that I think they planned on folks using OMV, a hand created RAID setup would confuse the script that updates the OLED.

There is no option but to use a hand created RAID setup as OMV doesn’t support creating RAIDs with disks attached via USB. Even it I was using OMV I would need to setup the RAID by hand. So I don’t think that’s the issue.

To clarify I think the OLED script is reporting the RAW space utilization which is indeed 100% as the entire disk is used for the RAID, but that isn’t really useful here.

Didn’t say it was helpful…

1 Like

The script is simply reporting the raid layout by parsing /proc/mdstat… so it is reporting the fact that the drives are consumed 100%. Because they are. For RAID.

The script is not reporting the utilization of the file system… that was/is created on top of the RAID volume. The MD subsystem has ZERO knowledge of how that storage is being used, it just knows that all of it is going to a files system.

Each drive us 100% committed to RAID, and the created RAID volume is 100% committed to a file system. Only the file system knows what blocks are free and what are not free.

It’s how RAID works.


def argonsysinfo_listraid():
	hddlist = []
	outputlist = []
	# cat /proc/mdstat
	# multiple mdxx from mdstat
	# mdadm -D /dev/md1

	ramtotal = 0
	errorflag = False
	try:
		hddctr = 0
		tempfp = open("/proc/mdstat", "r")
		alllines = tempfp.readlines()
		for temp in alllines:
			temp = temp.replace('\t', ' ')
			temp = temp.strip()
			while temp.find("  ") >= 0:
				temp = temp.replace("  ", " ")
			infolist = temp.split(" ")
			if len(infolist) >= 4:

				# Check if raid info
				if infolist[0] != "Personalities" and infolist[1] == ":":
					devname = infolist[0]
					raidtype = infolist[3]
					raidstatus = infolist[2]
					hddctr = 4
					while hddctr < len(infolist):
						tmpdevname = infolist[hddctr]
						tmpidx = tmpdevname.find("[")
						if tmpidx >= 0:
							tmpdevname = tmpdevname[0:tmpidx]
						hddlist.append(tmpdevname)
						hddctr = hddctr + 1
					devdetail = argonsysinfo_getraiddetail(devname)
					outputlist.append({"title": devname, "value": raidtype, "info": devdetail})

		tempfp.close()
	except IOError:
		# No raid
		errorflag = True

	return {"raidlist": outputlist, "hddlist": hddlist}
2 Likes

I have the same setup + OMV installed and my OLED panel reports the same. I understand the sentiments above for the “that’s how raid works, it should be 100%” thought but I think utility-wise, the OLED should be parsing raid capacity and not allocated drive capacity.

Another thing to note, I have my SSDs configured as RAID0 and see the correct reporting on the OLED, which I’d expect since it’s a merged FS. My HDDs are RAID1 and those report 100% used, which makes sense in theory since the drives are always mirroring each other so thus are both fully allocated from Day 0.

Define correct reporting…

What are you seeing for RAID0 and what are you expecting for RAID1? It’s just a script so it can be modified, however the script is simply reporting from /proc/mdstat…

So I decided to do some more spelunking in the script: Apparently the script calls:

mdadm -D /dev/<device? for all raidsets created.

size is reported from "the “Array Size :” label, and used is from “Used Dev Size:”

So assuming md0

$ sudo mdadm -D /dev/md0
/dev/md0:
           Version : 1.2
     Creation Time : Sat Feb  5 19:44:44 2022
        Raid Level : raid0
        Array Size : 2851750400 (2719.64 GiB 2920.19 GB)
      Raid Devices : 3
     Total Devices : 3
       Persistence : Superblock is persistent

       Update Time : Sat Feb  5 19:44:44 2022
             State : clean 
    Active Devices : 3
   Working Devices : 3
    Failed Devices : 0
     Spare Devices : 0

            Layout : original
        Chunk Size : 512K

Consistency Policy : none

              Name : pinas.xxx
              UUID : 0b1accdc:494a4114:8de5b0eb:f6b33898
            Events : 0

    Number   Major   Minor   RaidDevice State

and RAID1

sudo mdadm -D /dev/md0
/dev/md0:
           Version : 1.2
     Creation Time : Sat Feb  5 20:28:45 2022
        Raid Level : raid1
        Array Size : 937560384 (894.13 GiB 960.06 GB)
     Used Dev Size : 937560384 (894.13 GiB 960.06 GB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Sat Feb  5 20:29:21 2022
             State : clean, resyncing 
    Active Devices : 2
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 0

Consistency Policy : bitmap

     Resync Status : 0% complete

              Name : pinas.
              UUID : e0f3a2ae:cd802df6:5319c76c:01d13669
            Events : 7

    Number   Major   Minor   RaidDevice State
       0       8       16        0      active sync   /dev/sdb
       1       8       32        1      active sync   /dev/sdc

There is no “Used Dev Size:” for RAID0. This leaves the used size as zero. Given the meaning is “Amount of space used on the device for RAID” this value is kind of useless.

Would be nice it the raid page presented state (good/rebuilding/mirroring etc) and ignored things that don’t change (like the amount of RAID disk space utilized, etc).

Think I’ll be hacking some scripts.

2 Likes

@mrmsquared

Here is a simple two line fix for the storage utilization numbers:

diff --git a/argonsysinfo.py b/argonsysinfo.py
index 0e7e653..e674da7 100644
--- a/argonsysinfo.py
+++ b/argonsysinfo.py
@@ -176,7 +176,7 @@ def argonsysinfo_listhddusage():
        raidctr = 0
        while raidctr < len(raidlist['raidlist']):
                raiddevlist.append(raidlist['raidlist'][raidctr]['title'])
-               outputobj[raidlist['raidlist'][raidctr]['title']] = {"used":int(raidlist['raidlist'][raidctr]['info']['used']), "total":int(raidlist['raidlist'][raidctr]['info']['size'])}
+               #outputobj[raidlist['raidlist'][raidctr]['title']] = {"used":int(raidlist['raidlist'][raidctr]['info']['used']), "total":int(raidlist['raidlist'][raidctr]['info']['size'])}
                raidctr = raidctr + 1

        rootdev = argonsysinfo_getrootdev()
@@ -207,7 +207,7 @@ def argonsysinfo_listhddusage():
                        if curdev in raidlist['hddlist']:
                                continue
                        elif curdev in raiddevlist:
-                               continue
+                               curdev = curdev
                        elif curdev[0:2] == "sd" or curdev[0:2] == "hd":
                                curdev = curdev[0:-1]
                        else:
@@ -325,4 +325,4 @@ def argonsysinfo_getraiddetail(devname):
                                failed = infolist[1]
                        elif infolist[0].lower() == "spare devices":
                                spare = infolist[1]
-       return {"state": state, "raidtype": raidtype, "size": int(size), "used": int(used), "devices": int(total), "active": int(active), "working": int(working), "failed": int(failed), "spare": int(spare)}
\ No newline at end of file
+       return {"state": state, "raidtype": raidtype, "size": int(size), "used": int(used), "devices": int(total), "active": int(active), "working": int(working), "failed": int(failed), "spare": int(spare)}

Hand edit remembering to preserve the tabs… Why anyone would use tabs in python is beyond me.

Or copy the above to file.patch

$ sudo -i
$ systemctl stop argononed.service
$ cd /etc/argon/
$ cp argonsysinfo.py argonsysinfo.py_old
$ patch -p1 < file.patch
$ systemctl start argononed.service
$ systemctl status argononed.service

The storage page show now show are reasonably accurate number.

4 Likes

@NHHiker thank you, this fixed the ‘Storage Utilization’ page.

I also wanted to see the correction within the individual RAID Details pages. I’ve gone ahead and modified the sysinfo script further to provide more clarity on those pages by utilizing the ‘Used Size’ from df instead of mdadm.

#!/usr/bin/python3

#
# Misc methods to retrieve system information.
#

import os
import re
import time
import socket

def argonsysinfo_listcpuusage(sleepsec = 1):
	outputlist = []
	curusage_a = argonsysinfo_getcpuusagesnapshot()
	time.sleep(sleepsec)
	curusage_b = argonsysinfo_getcpuusagesnapshot()

	for cpuname in curusage_a:
		if cpuname == "cpu":
			continue
		if curusage_a[cpuname]["total"] == curusage_b[cpuname]["total"]:
			outputlist.append({"title": cpuname, "value": "0%"})
		else:
			total = curusage_b[cpuname]["total"]-curusage_a[cpuname]["total"]
			idle = curusage_b[cpuname]["idle"]-curusage_a[cpuname]["idle"]
			outputlist.append({"title": cpuname, "value": int(100*(total-idle)/(total))})
	return outputlist

def argonsysinfo_getcpuusagesnapshot():
	cpupercent = {}
	errorflag = False
	try:
		cpuctr = 0
		# user, nice, system, idle, iowait, irc, softirq, steal, guest, guest nice
		tempfp = open("/proc/stat", "r")
		alllines = tempfp.readlines()
		for temp in alllines:
			temp = temp.replace('\t', ' ')
			temp = temp.strip()
			while temp.find("  ") >= 0:
				temp = temp.replace("  ", " ")
			if len(temp) < 3:
				cpuctr = cpuctr +1
				continue

			checkname = temp[0:3]
			if checkname == "cpu":
				infolist = temp.split(" ")
				idle = 0
				total = 0
				colctr = 1
				while colctr < len(infolist):
					curval = int(infolist[colctr])
					if colctr == 4 or colctr == 5:
						idle = idle + curval
					total = total + curval
					colctr = colctr + 1
				if total > 0:
					cpupercent[infolist[0]] = {"total": total, "idle": idle}
			cpuctr = cpuctr +1

		tempfp.close()
	except IOError:
		errorflag = True
	return cpupercent


def argonsysinfo_liststoragetotal():
	outputlist = []
	ramtotal = 0
	errorflag = False

	try:
		hddctr = 0
		tempfp = open("/proc/partitions", "r")
		alllines = tempfp.readlines()

		for temp in alllines:
			temp = temp.replace('\t', ' ')
			temp = temp.strip()
			while temp.find("  ") >= 0:
				temp = temp.replace("  ", " ")
			infolist = temp.split(" ")
			if len(infolist) >= 4:
				# Check if header
				if infolist[3] != "name":
					parttype = infolist[3][0:3]
					if parttype == "ram":
						ramtotal = ramtotal + int(infolist[2])
					elif parttype[0:2] == "sd" or parttype[0:2] == "hd":
						lastchar = infolist[3][-1]
						if lastchar.isdigit() == False:
							outputlist.append({"title": infolist[3], "value": argonsysinfo_kbstr(int(infolist[2]))})
					else:
						# SD Cards
						lastchar = infolist[3][-2]
						if lastchar[0] != "p":
							outputlist.append({"title": infolist[3], "value": argonsysinfo_kbstr(int(infolist[2]))})

		tempfp.close()
		#outputlist.append({"title": "ram", "value": argonsysinfo_kbstr(ramtotal)})
	except IOError:
		errorflag = True
	return outputlist

def argonsysinfo_getram():
	totalram = 0
	totalfree = 0
	tempfp = open("/proc/meminfo", "r")
	alllines = tempfp.readlines()

	for temp in alllines:
		temp = temp.replace('\t', ' ')
		temp = temp.strip()
		while temp.find("  ") >= 0:
			temp = temp.replace("  ", " ")
		infolist = temp.split(" ")
		if len(infolist) >= 2:
			if infolist[0] == "MemTotal:":
				totalram = int(infolist[1])
			elif infolist[0] == "MemFree:":
				totalfree = totalfree + int(infolist[1])
			elif infolist[0] == "Buffers:":
				totalfree = totalfree + int(infolist[1])
			elif infolist[0] == "Cached:":
				totalfree = totalfree + int(infolist[1])
	if totalram == 0:
		return "0%"
	return [str(int(100*totalfree/totalram))+"%", str((totalram+512*1024)>>20)+"GB"]


def argonsysinfo_gettemp():
	try:
		tempfp = open("/sys/class/thermal/thermal_zone0/temp", "r")
		temp = tempfp.readline()
		tempfp.close()
		return float(int(temp)/1000)
	except IOError:
		return 0
	cval = val/1000
	fval = 32+9*val/5000

def argonsysinfo_getip():
	ipaddr = ""
	st = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
	try: 
		# Connect to nonexistent device
		st.connect(('254.255.255.255', 1))
		ipaddr = st.getsockname()[0]
	except Exception:
		ipaddr = 'N/A'
	finally:
		st.close()
	return ipaddr


def argonsysinfo_getrootdev():
	tmp = os.popen('mount').read()
	alllines = tmp.split("\n")

	for temp in alllines:
		temp = temp.replace('\t', ' ')
		temp = temp.strip()
		while temp.find("  ") >= 0:
			temp = temp.replace("  ", " ")
		infolist = temp.split(" ")
		if len(infolist) >= 3:

			if infolist[2] == "/":
				return infolist[0]
	return ""

def argonsysinfo_listhddusage():
	outputobj = {}
	raidlist = argonsysinfo_listraid()
	raiddevlist = []
	raidctr = 0
	while raidctr < len(raidlist['raidlist']):
		raiddevlist.append(raidlist['raidlist'][raidctr]['title'])
		#outputobj[raidlist['raidlist'][raidctr]['title']] = {"used":int(raidlist['raidlist'][raidctr]['info']['used']), "total":int(raidlist['raidlist'][raidctr]['info']['size'])}
		raidctr = raidctr + 1

	rootdev = argonsysinfo_getrootdev()

	tmp = os.popen('df').read()
	alllines = tmp.split("\n")

	for temp in alllines:
		temp = temp.replace('\t', ' ')
		temp = temp.strip()
		while temp.find("  ") >= 0:
			temp = temp.replace("  ", " ")
		infolist = temp.split(" ")
		if len(infolist) >= 6:
			if infolist[1] == "Size":
				continue
			if len(infolist[0]) < 5:
				continue
			elif infolist[0][0:5] != "/dev/":
				continue
			curdev = infolist[0]
			if curdev == "/dev/root" and rootdev != "":
				curdev = rootdev
			tmpidx = curdev.rfind("/")
			if tmpidx >= 0:
				curdev = curdev[tmpidx+1:]

			if curdev in raidlist['hddlist']:
				continue
			elif curdev in raiddevlist:
				curdev = curdev
			elif curdev[0:2] == "sd" or curdev[0:2] == "hd":
				curdev = curdev[0:-1]
			else:
				curdev = curdev[0:-2]
			if curdev in outputobj:
				outputobj[curdev] = {"used":outputobj[curdev]['used']+int(infolist[2]), "total":outputobj[curdev]['total']+int(infolist[1])}
			else:
				outputobj[curdev] = {"used":int(infolist[2]), "total":int(infolist[1])}

	return outputobj

def argonsysinfo_kbstr(kbval, wholenumbers = True):
	remainder = 0
	suffixidx = 0
	suffixlist = ["KB", "MB", "GB", "TB"]
	while kbval > 1023 and suffixidx < len(suffixlist):
		remainder = kbval & 1023
		kbval  = kbval >> 10
		suffixidx = suffixidx + 1

	#return str(kbval)+"."+str(remainder) + suffixlist[suffixidx]
	remainderstr = ""
	if kbval < 100 and wholenumbers == False:
		remainder = int((remainder+50)/100)
		if remainder > 0:
			remainderstr = "."+str(remainder)
	elif remainder >= 500:
		kbval = kbval + 1
	return str(kbval)+remainderstr + suffixlist[suffixidx]

def argonsysinfo_listraid():
	hddlist = []
	outputlist = []
	# cat /proc/mdstat
	# multiple mdxx from mdstat
	# mdadm -D /dev/md1

	ramtotal = 0
	errorflag = False
	try:
		hddctr = 0
		tempfp = open("/proc/mdstat", "r")
		alllines = tempfp.readlines()
		for temp in alllines:
			temp = temp.replace('\t', ' ')
			temp = temp.strip()
			while temp.find("  ") >= 0:
				temp = temp.replace("  ", " ")
			infolist = temp.split(" ")
			if len(infolist) >= 4:

				# Check if raid info
				if infolist[0] != "Personalities" and infolist[1] == ":":
					devname = infolist[0]
					raidtype = infolist[3]
					raidstatus = infolist[2]
					hddctr = 4
					while hddctr < len(infolist):
						tmpdevname = infolist[hddctr]
						tmpidx = tmpdevname.find("[")
						if tmpidx >= 0:
							tmpdevname = tmpdevname[0:tmpidx]
						hddlist.append(tmpdevname)
						hddctr = hddctr + 1
					devdetail = argonsysinfo_getraiddetail(devname)
					outputlist.append({"title": devname, "value": raidtype, "info": devdetail})

		tempfp.close()
	except IOError:
		# No raid
		errorflag = True

	return {"raidlist": outputlist, "hddlist": hddlist}


def argonsysinfo_getraiddetail(devname):
        state = ""
        raidtype = ""
        size = 0
        used = 0
        total = 0
        working = 0
        active = 0
        failed = 0
        spare = 0
        tmp = os.popen('mdadm -D /dev/'+devname).read()
        alllines = tmp.split("\n")

        # Used Size Fix
        tmp_used = os.popen('df /dev/'+devname).read().split("\n")[1].replace(" ", "_")
        used_size = re.sub(r'(.)\1+', r'\1', tmp_used)
        x = used_size.find("_")
        y = used_size.find("_", x+1)
        z = used_size.find("_", y+1)
        used = used_size[y+1:z]

        for temp in alllines:
                temp = temp.replace('\t', ' ')
                temp = temp.strip()
                while temp.find("  ") >= 0:
                        temp = temp.replace("  ", " ")
                infolist = temp.split(" : ")
                if len(infolist) == 2:
                        if infolist[0].lower() == "raid level":
                                raidtype = infolist[1]
                        elif infolist[0].lower() == "array size":
                                tmpidx = infolist[1].find(" ")
                                if tmpidx > 0:
                                        size = (infolist[1][0:tmpidx])
                        #elif infolist[0].lower() == "used dev size":
                        #        tmpidx = infolist[1].find(" ")
                        #        if tmpidx > 0:
                        #                used = (infolist[1][0:tmpidx])
                        elif infolist[0].lower() == "state":
                                state = infolist[1]
                        elif infolist[0].lower() == "total devices":
                                total = infolist[1]
                        elif infolist[0].lower() == "active devices":
                                active = infolist[1]
                        elif infolist[0].lower() == "working devices":
                                working = infolist[1]
                        elif infolist[0].lower() == "failed devices":
                                failed = infolist[1]
                        elif infolist[0].lower() == "spare devices":
                                spare = infolist[1]
        return {"state": state, "raidtype": raidtype, "size": int(size), "used": int(used), "devices": int(total), "active": int(active), "working": int(working), "failed": int(failed), "spare": int(spare)}
2 Likes

Thanks @NHHiker @mrmsquared, applying these changes got both the storage utilization and RAID pages showing the correct usage!

1 Like

No problem! It was a quick and easy fix.

I was thinking about changing the raid page to show the state of raid. Was thinking of showing the state of the md device itself, with rebuild, mirroring etc. Leave the utilization to the storage page, and make the raid page more about raid…

Why have two pages with similar info? i.e. usage?

I was considering a git repository for the changes…

3 Likes

I agree, duplicate info right now among those two pages. Let me know if you get around to making it more raid-y

Also, I put my tweaks in a repl: https://replit.com/@mrmsquared/ArgonEonTweaks#argonsysinfo.py

A git repo sounds like a very good idea!

1 Like