[LFCS] Managing Software RAID

mdadm is a super cool command in Linux used to manage MD devices aka Linux Software RAID. Before we jump in, let’s see what is RAID. – –Redunant array of independent disks
–if disk gets corrupted, then data loss
–using RAID, if one disk fails, other will take over

$ man page says this,

RAID devices are virtual devices created from two or more real block devices. This allows multiple devices (typically disk drives or partitions thereof) to be combined into a single device to hold (for example) a single filesystem. Some RAID levels include redundancy and so can survive some degree of device failure.

Understanding RAID soln,
RAID O- Striping { one big device based on multiple disk, no redundancy or easy recovery}
RAID 1- Mirroring { 2 disks, identical }
RAID 5- striping with distributed parity { if data is written with parity info, if one disks fails, then restore the data }
RAID 6- striping with dual distributed parity { redundant parity is written,advancement of RAID 5 }
RAID 10- mirrored and striped { minimum of 10 disks, 2 for striping, 2 for mirrored}

Sample question: How to create a RAID 5 Device using 3 disk device of 1 GB each. Also allocate additional device as spare device.
— Put a file system on it and mount it on /raid
— Fail one of the devices, monitor what is happening
— Replace the failed device with spare device

Solution:

$ cat /proc/partitions
$ fdisk -l { list the partition tables for the device, if no device specified then list all the partitions from the system }
$ fdisk /dev/sdc
-n create a new partitions { size as +1G }
-m {help}
-t :L {enter “fd” for Linux raid auto}
-w { write the entries to persist }
$ partprobe { inform the OS partition table changes }
$ vim /etc/fstab { before we proceed, let’s verify the disks are not used for any mounting. In my case, I had used as swap device mounting so got an error saying device is busy error. Rmv the entry, reboot }
$ mdadm –create /dev/md1 -l 5 -x 1 –raid-disk=3 /dev/sdc1 /dev/sdc2 /dev/sdc3 /dev/sdc4 –verbose –auto=yes
$ mdadm –detail /dev/md1 { list details after creation, should see 3 device + 1 spare device }

$ mdadm —fail dev/md1 /dev/sdc1 { to simulate the failure }
$ mdadm –remove /dev/md1 /dev/sdc1 { remove the faulty one }
$ mdadm –add /dev/md1 /dev/sdc1 { add the device back to the pool as spare device if healthy }

other disk related commands,
$ blkid $ blkid /dev/sdc
$ df -h, df -h -T, df -hT /home
$ du -h /home, du -sh /home/mydir
$ mount /dev/sdc5 /mnt, cd /mnt , touch file1 { after mounting make entry in /etc/fstab to persist}
$ mount -a { to mount all fs mentioned in fstab}
$ mkfs.ext4 /dev/sda4 { format a partition of type ext4, after creating a partition }

Command output:

root@mikky100:~# mdadm –fail /dev/md1 /dev/sdc1 { Simulate the failure }
mdadm: set /dev/sdc1 faulty in /dev/md1

root@mikky100:~# mdadm –detail /dev/md1 { view the detail after the failure, we should see the spare disk getting rebuild }
/dev/md1:
Version : 1.2
Creation Time : Mon Jun 11 06:10:34 2018
Raid Level : raid5
Array Size : 1951744 (1906.32 MiB 1998.59 MB)
Used Dev Size : 975872 (953.16 MiB 999.29 MB)
Raid Devices : 3
Total Devices : 4
Persistence : Superblock is persistent

Update Time : Mon Jun 11 17:06:09 2018
State : clean, degraded, recovering
Active Devices : 2
Working Devices : 3
Failed Devices : 1
Spare Devices : 1

Layout : left-symmetric
Chunk Size : 512K

Rebuild Status : 3% complete

Name : mikky100:1 (local to host mikky100)
UUID : 772f743c:b1209727:6910411d:690d6294
Events : 20

Number Major Minor RaidDevice State
3 8 36 0 spare rebuilding /dev/sdc4
1 8 34 1 active sync /dev/sdc2
4 8 35 2 active sync /dev/sdc3

0 8 33 – faulty /dev/sdc1

root@mikky100:~# mdadm –detail /dev/md1
/dev/md1:
Version : 1.2
Creation Time : Mon Jun 11 06:10:34 2018
Raid Level : raid5
Array Size : 1951744 (1906.32 MiB 1998.59 MB)
Used Dev Size : 975872 (953.16 MiB 999.29 MB)
Raid Devices : 3
Total Devices : 4
Persistence : Superblock is persistent

Update Time : Mon Jun 11 17:08:13 2018
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 1
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 512K

Name : mikky100:1 (local to host mikky100)
UUID : 772f743c:b1209727:6910411d:690d6294
Events : 37

Number Major Minor RaidDevice State
3 8 36 0 active sync /dev/sdc4
1 8 34 1 active sync /dev/sdc2
4 8 35 2 active sync /dev/sdc3

0 8 33 – faulty /dev/sdc1

root@mikky100:~# mdadm –add /dev/md1 /dev/sdc1 { add the disk back as spare }

root@mikky100:~# mdadm –detail /dev/md1
/dev/md1:
Version : 1.2
Creation Time : Mon Jun 11 06:10:34 2018
Raid Level : raid5
Array Size : 1951744 (1906.32 MiB 1998.59 MB)
Used Dev Size : 975872 (953.16 MiB 999.29 MB)
Raid Devices : 3
Total Devices : 4
Persistence : Superblock is persistent

Update Time : Mon Jun 11 17:12:21 2018
State : clean
Active Devices : 3
Working Devices : 4
Failed Devices : 0
Spare Devices : 1

Layout : left-symmetric
Chunk Size : 512K

Name : mikky100:1 (local to host mikky100)
UUID : 772f743c:b1209727:6910411d:690d6294
Events : 39

Number Major Minor RaidDevice State
3 8 36 0 active sync /dev/sdc4
1 8 34 1 active sync /dev/sdc2
4 8 35 2 active sync /dev/sdc3

5 8 33 – spare /dev/sdc1

WIIFM-“Powershell available on Linux”

When I first read this announcement – https://azure.microsoft.com/en-us/blog/powershell-is-open-sourced-and-is-available-on-linux/, I thought what the heck we are trying to solve when “Bash/Python/Perl” Linux already has got for automation & scripting. But it is proved wrong after this video – https://youtu.be/2WZwv7TxqZ0

But once again, this is a very cool move from our Powershell team by opening up the source code + extending support to Linux and MacOS. Devops would love this for sure, it comes handy for managing or automating Azure resources across the OS. They are plans to include most of the modules down the line so that, seamless execution for sure.

Keytakeaway:- If we have any PS script written in Windows say Azure VM creation or management /Dockers/AWS storage/VMware resource management etc then we can reuse the same script in Linux & Macos. There is no difference in the syntax. Just copy paste should work without any difference.

Sweet – “We are partnering with third party companies – Chef, Amazon Web Services, VMware, and Google to name a few – to create a rich, seamless experience across the platforms you know and use.”

Write once, runs any ware philosophy –heterogeneous management toolset.

https://channel9.msdn.com/Blogs/hybrid-it-management/PowerShell-on-Linux-and-Open-Source/player

Issue with Android Emulator running under Hyper-V (VM’s)

Recently I prepared a VM in our lab for checking an issue repro. The issue was with Android Studio running(Java code) hitting Azure Storage blobs throwing an exception for some weird reason. So quickly jumped in creating a Lab machine and installed all the software like Android Studio, ton of updates and 90+ packages having various emulators. It took quite sometime to install everything(tiring), fix the Java path(this is pain and behaves differently between editors) etc.

After setting up all, I ran the code for the first time happily but got this error which brought everything to halt. After bit of research, learned that we cannot run phone emulators within a Hyper-Vised (VM‘s), looks like emulator runs under another Hyper-V which means running Hyper-V in another Hyper-V not supported.  

Solution:- We may have to use our own laptop/physical machine connecting to phone directly with USB cable . This gives good experience rather emulator way. this is also faster when it comes to debugging or disconnect or redeploy or tracing end to end.

Let me know if there is a way to mitigate this along with your dev setup experience. Any tips and tricks also welcome..

Happy learning !

How to RDP into Azure Linux machine using Xrdp

I ran into this issue after creating Azure Linux VM and attempting to configure RDP for the first time. At Linux side, I have installed Xrdp and configured for RDP access. But when I try to RDP from my windows machine then it failed. I spent few hours troubleshooting this issue without clue but later it turned out to be a “port” issue. Yes, by default Azure VM’s will expose or allow very limited port say 80,443 for Windows and in Linux Port 22 for SSH connection. So in order to enable RDP for Linux/Win machines, then you may have to add this port 3389 under “incoming security rules” as below.

I followed this blog post for xrdp setup up but nothing specific about Azure port as such. So thought of summarizing my learning here in steps to follow quickly for achieving RDP access to Azure Linux VM’s.

1) Login to new portal (portal.azure.com), Click “+ New” for adding resource, then select Virtual machine. You will find Ubuntu server OS list. Let say pick “Ubuntu Server 14.04.4 LTS”. Make sure to note down the Username, Password and Public IP(PIP) for later reference.

2) Once VM has been created, use Putty.exe to connect to the VM over SSH(Port 22) which is enabled by default. Enter the PIP and hit the connect button. You will get a console asking to enter username and password. Once provided, you will land in Linux terminal where we are going to execute the next set of commands in order.

3) Command to Install Desktop and xRDP at Linux VM side.

$ sudo apt-get update

$ sudo apt-get install ubuntu-desktop
$ sudo apt-get install xfce4
$ sudo apt-get install xrdp
$ echo xfce4-session >~/.xsession
$ sudo nano /etc/xrdp/startwm.sh  <attention to the last line, it should be exactly like this>
#!/bin/sh
if [ -r /etc/default/locale ]; then
. /etc/default/locale
export LANG LANGUAGE
fi
startxfce4
$ sudo service xrdp restart

4) Now go back to Azure portal and Add Port 3389 under Inbound security rule (as in the above screenshot). P.s:- 3389 is well known port used by Azure for RDP.

5) Now from Windows machine, Launch “mstsc” to input VIP and username & password to Xrdp.

 

 

 

 

 

 

P.s:- Clipboard copy not supported here.

I missed the step 4 and wasted quite a lot of time figuring out.

Hope this will save someone’s time 🙂 .

<update:7/21>
               a. Finding the inbound rules is little tricky..
                       VM > settings > network interfaces >settings > NSG > select the NSG > settings > inbound rules
               b. lately noticed our official article on this https://azure.microsoft.com/en-in/documentation/articles/virtual-machines-linux-classic-remote-desktop/

</update:7/21>

Update:8/1 – tips

1) exit ->logout, 2) sudo –  -> (to get into root) 3) sudo passwd root

4) download any .deb (tar xzvf file.tar.gz) extract, check for bin > say check for "sh" file -pycharm.sh, right click > execute

 

<Unsupported: 9/28/2016>

Unsupported: How to install xRDP for GUI remote access on Linux
https://blogs.msdn.microsoft.com/linuxonazure/2016/09/26/unsupported-how-to-install-xrdp-for-gui-remote-access-on-linux/

</Unsupported>

.NET today, tomorrow and future..

This blog post was drafted few weeks ago but stuck in the same state because I was not sure what to add or write on top of what we learned from our recent dotnetConf2016.  Any how today I decided to wrap this so that it will be useful for readers. This post is mainly written for .NET developer who wanted to know where we are heading as  a community.

 

Motivation:- If you notice in recent times, there is a bunch of articles targeting .NET platform and its future. Few trolled about WPF and Winform roadmap. I saw most of them are one sided and simply trolled comparing with recent programming stack like LAMP/MEAN as an alternate. But the true story is, .NET is not going to go away. This applies to Java as well. I have my good old friends still doing WPF consulting work for big banking companies and happy with that. We all need “Growth Mindset” in looking at them.

We may have to agree that our recent startup companies are going 100% open source but when it comes to desktop development still WPF rules.The skill what you acquired from WPF is not going to be waste, because UWP is based on XAML. .NET is growing and evolving faster than ever. If you look at our .NET Foundation, it has got some fantastic open source projects run by community. We are breaking the platform dependencies, IIS, underlying win32 thick references which is the great thing for cross platform movement.  

 

Technology:- Things are changing very rapidly when it comes to programming side especially for the web & mobile technologies. As we march towards “mobile first cloud first” world, it makes sense to investment towards platform mobility and in cloud. please note, “mobile first” here is nothing to do with our mobile device. It’s about application mobility – should be able to port our app across the device.

Every week/month we see new set of tools/services/JS framework/scripting things comes out constantly and occupies the head line or top slot in tech conferences. One shouldn’t be worrying about new things, we should stay calm and keep learning at least one new languages every year as an investment. If I remember correctly, for last few years we are seeing a trend in IT requirement. Yr 2013 was for Big data, 2014 for Machine learning/deep learning/AI , 2015 for IoT + dockers,  again in 2016 seeing BoT, headless servers, nano servers, cognitive services and what not. Btw, Cloud and OSS is the constant thing for last few years and going to be more talked in coming years as well . It shouldn’t be a matter for us to pick up this in couple of weekends if we know the fundamentals of programming & design standards.

 

 

Roadmap:- If we know the technology roadmap in advance, then it is easy for anyone to refer and suggest for customers. I had this .NET roadmap guide constantly referred, forwarded to architects for newer design consideration but the last one published was in the year 2013 and sadly no update after that.

Get our last guide “.NET Technology Guide for business applications” downloaded from- > here

[P.s: Some of them are already outdated]

 

 

 

Few months back, I reached out to Cesar(author of this PDF) seeking clarity on our guidance update. He has promptly shared the preview version but today I see the same guide available for download freely here. What a timing

Microsoft Platform and Tools for Mobile App Development –

https://blogs.msdn.microsoft.com/microsoft_press/2016/06/22/free-ebook-microsoft-platform-and-tools-for-mobile-app-development/

 

 

Having said all, Thanks to our recent .net Conference for sharing more clarity around .NET future and developments. If you are .NET developer and not seen the updates from our recent dotnetConf 2016 –  then you are in the right place to get the updates through screenshot.

As title says, this is our current landscape. If you look closer, there is “no code sharing” across the platform and also carry the same learning. The new addition to this slide is our “UWP” and “Xamarin” story.

 

 

How about .NET tomorrow:- Yes, it allows us to reuse the skills by mastering one/two library not a platform. You can build and reuse the same code across the platform say iOS or Android or to OSX.

Main objective here is to make the “.NET standard library” to grow without updating platforms. It is going to be decoupled so that you can update libraries as you wish.

 

 

How is .NET Future looks like: More and more “.NET standard” supported API’s are expected in few months. Complex Libs like App Domains, Full Reflection and Binary Serialization are also getting considered. In near future, one should be able to easily move the code b/w .NET framework or .NET Core or .Xamarin.

 

 

.NET future- sub slide (.NET Framework) :-

a) C# – C#7, VB 15 is the talk of the town

b) Going forward- new version of windows 10,

c) Winforms/WPF  –> Store model

d) Winform/WPF convert to Store apps model, use store to push across the organization

 

 

What others are busy learning..

 

 

Closing note: & Key take away:

 

 

 

 

Recommended video:-

.NET Conf Day 1 Keynote – Scott Hunter

.NET Conf Day 2 Keynote – Miguel de Icaza

.NET Conf Day 3 Keynote – Scott Hanselman on The State of .NET

 

Watch the event videos here:-https://channel9.msdn.com/Events/dotnetConf/2016

 

Hope this is useful for you..

<update : 6/28/2016>

Another milestone on developer’s world: .NET Core & ASP.NET Core 1.0 got released at Red Hat DevNation. https://channel9.msdn.com/Events/Linux/DevNation-2016

.NET Home Page http://dot.net/

.NET Documentation https://docs.microsoft.com/en-us/dotnet/

Announcement https://blogs.msdn.microsoft.com/dotnet/2016/06/27/announcing-net-core-1-0/

</update>