experchange > solaris

YTC#1 (12-19-18, 11:47 AM)
Before my (slow, painful) upgrade to 11.4, my immutable zone worked fine.

Now, it fails as per below, unless I set file-mac-profile=none

---8<

zonecfg -z ytcadmin export

create -b

set brand=solaris

set zonepath=/zones/ytcadmin

set autoboot=true

set file-mac-profile=fixed-configuration

add anet

set linkname=net0

set lower-link=net1

set configure-allowed-address=true

set mac-address=random

end

root@ytc1-s:~# zoneadm -z ytcadmin halt

root@ytc1-s:~# zoneadm -z ytcadmin boot

zone 'ytcadmin': error: mount of /system/volatile failed: Permission denied

zone 'ytcadmin': error: unable to mount filesystems

zone 'ytcadmin': cannot unmount '/zones/ytcadmin/root': Device busy

zone 'ytcadmin': ERROR: unable to unmount /zones/ytcadmin/root.

zone 'ytcadmin': ERROR: Unable to mount zone root dataset.

zoneadm: zone ytcadmin: call to zoneadmd(8) failed: zoneadmd(8) returned
an error 1 (unspecified error)

---8<

What gives ?
Casper H.S. Dik (12-19-18, 05:30 PM)
YTC#1 <bdp> writes:

>Before my (slow, painful) upgrade to 11.4, my immutable zone worked fine.


>Now, it fails as per below, unless I set file-mac-profile=none


>root@ytc1-s:~# zoneadm -z ytcadmin halt


>root@ytc1-s:~# zoneadm -z ytcadmin boot


>zone 'ytcadmin': error: mount of /system/volatile failed: Permission denied


>zone 'ytcadmin': error: unable to mount filesystems


>zone 'ytcadmin': cannot unmount '/zones/ytcadmin/root': Device busy


>zone 'ytcadmin': ERROR: unable to unmount /zones/ytcadmin/root.


>zone 'ytcadmin': ERROR: Unable to mount zone root dataset.


>zoneadm: zone ytcadmin: call to zoneadmd(8) failed: zoneadmd(8) returned
>an error 1 (unspecified error)


>---8<


>What gives ?


Which version of Solaris and which file-mac-profile?

Are there files in the mount points? E.g., anything under /system/volatile
in the zone?

What are the privileges of the current process?

Casper
YTC#1 (12-19-18, 07:06 PM)
On 19/12/2018 15:30, Casper H.S. Dik wrote:
> YTC#1 <bdp> writes:
> Which version of Solaris and which file-mac-profile?


From S11.3 SR35 -> S11.4 SRU4
> Are there files in the mount points? E.g., anything under /system/volatile
> in the zone?

Well, yes, the system creates them when it boots.
And as /system/volatile is mounted on swap, as I understand it that is
created boot time ?

ls -alrt
total 5249
drwxr-xr-x 5 root root 5 May 2 2018 ..
Drw------- 1 pkg5srv pkg5srv 0 Dec 19 10:23 zoneproxy_door
Drw-r--r-- 1 dladm netadm 0 Dec 19 10:23
repository_door.libnetcfg.23
drwxrwxr-x 2 dladm netadm 185 Dec 19 10:23 dladm
Drw-r--r-- 1 root root 0 Dec 19 10:23 zonestat_door
Drw-r--r-- 1 root root 0 Dec 19 10:23 repository_door
-rw------- 1 root root 0 Dec 19 10:23 ipsecconf.lock
-rw-r--r-- 1 root root 4096 Dec 19 10:23 tzsync
Dr--r--r-- 1 root root 0 Dec 19 10:23 sysobjdb_door
drwx------ 2 root root 185 Dec 19 10:23 critpid
-rw------- 1 root root 0 Dec 19 10:23 init.GNXAAc
drwxr-xr-x 2 netadm netadm 323 Dec 19 10:23 in.ndpd
drwxr-xr-x 5 root root 304 Dec 19 10:23 opengl
drwxr-xr-x 2 root root 251 Dec 19 10:23 dbus
drwxr-xr-x 2 root root 405 Dec 19 10:23 rad
drwxr-xr-x 2 netadm netadm 252 Dec 19 10:23 netcfg
drwxr-xr-x 2 netadm netadm 254 Dec 19 10:23 ipadm
Dr--r--r-- 1 root root 0 Dec 19 10:23 name_service_door
-rw-r--r-- 1 root root 524296 Dec 19 10:23 audit_cache.bin
-rw-r--r-- 1 root root 5 Dec 19 10:23 sshd.pid
prw------- 1 root root 0 Dec 19 10:23 cronfifo
drwxr-xr-x 2 daemon daemon 117 Dec 19 10:23 daemon
-rw-r--r-- 1 root root 5 Dec 19 10:23 filesystem-autofs.lock
srwxr-xr-x 1 root root 0 Dec 19 10:23 inetd.uds
drwxr-xr-x 3 root root 181 Dec 19 10:23 svc:
drwxrwxr-x 2 root daemon 186 Dec 19 10:23 rpc_door
drwxr-xr-x 2 root root 190 Dec 19 10:23 webui
srwx------ 1 webservd root 0 Dec 19 10:23
webui-wsgi.2784.0.1.sock
Drw-rw-r-- 1 root root 0 Dec 19 10:23 sstore_door
Drw-r--r-- 1 root root 0 Dec 19 10:23 syslog_door
-rw-r--r-- 1 root root 5 Dec 19 10:23 syslog.pid
prw------- 1 root root 0 Dec 19 10:23 startdpipe
drwxr-xr-x 2 root root 195 Dec 19 10:23 zones
drwxr-xr-x 3 root root 191 Dec 19 10:23 sysevent_channels
-rw------- 1 root root 1474560 Dec 19 10:23 svc_nonpersist.db
-rw------- 1 root root 524928 Dec 19 10:23
svc_nonpersist.db-journal
-rw------- 1 root smmsp 38 Dec 19 10:24 sendmail.pid
prw------- 1 root root 0 Dec 19 17:01 initpipe
-rw------- 1 root root 656 Dec 19 17:01 init.state
drwxr-xr-x 17 root sys 2892 Dec 19 17:01 .
drwx------ 2 root root 54655 Dec 19 17:01 sstore
-rw-r--r-- 1 root bin 1860 Dec 19 17:01 utmpx
prw------- 1 root root 0 Dec 19 17:01 utmppipe
root@ytcadmin:/system/volatile# df -h .
Filesystem Size Used Available Capacity Mounted on
swap 7.5G 5.3M 7.5G 1% /system/volatile
root@ytcadmin:/system/volatile# last -3 reboot
reboot system boot Wed Dec 19 10:23
reboot system down Wed Dec 19 10:10
reboot system boot Wed Dec 19 09:38

> What are the privileges of the current process?


Which process ? zoneadm ?
ppriv 2292
2292: /usr/lib/zones/zoneadmd -z ytcadmin
flags = <none>
E: all
I: basic
P: all
L: all
YTC#1 (12-19-18, 10:38 PM)
On 19/12/2018 15:30, Casper H.S. Dik wrote:
> YTC#1 <bdp> writes:
> Which version of Solaris and which file-mac-profile?
> Are there files in the mount points? E.g., anything under /system/volatile
> in the zone?


Aha !
With the zone halted

---8<
root@ytc1-s:~# ls -al /zones/ytcadmin/root/system/volatile/
total 35
drwxr-xr-x 2 root sys 3 Feb 28 2013 .
drwxr-xr-x 5 root root 5 May 2 2018 ..
-rw------- 1 root root 0 Feb 28 2013 zoneproxy_door
---8<

So, I'd guess I have had some sort of bug for over 5 years, but still
managed to run as an immutable zone ? Anyway I've moved it away, reset
the file-mac-profile to fixed-configuration, and rebooted.

It now works.
Casper H.S. Dik (12-20-18, 09:47 AM)
YTC#1 <bdp> writes:

>From S11.3 SR35 -> S11.4 SRU4
>> Are there files in the mount points? E.g., anything under /system/volatile
>> in the zone?

>Well, yes, the system creates them when it boots.
>And as /system/volatile is mounted on swap, as I understand it that is
>created boot time ?


What is under the mount point? Immutable zones do not like mounting
in populated directory.

>root@ytcadmin:/system/volatile# df -h .
>Filesystem Size Used Available Capacity Mounted on
>swap 7.5G 5.3M 7.5G 1% /system/volatile


Try:

mkdir /tmp/root
mount -F lofs -o nosub / /tmp/root

and see what is available under /system/volatile.

Though it is somewhat strange because we generally allow it for
this particular directory (just checking on immutable global zone system
and there is some old stuff hanging around there.

Casper
Casper H.S. Dik (12-20-18, 10:05 AM)
YTC#1 <bdp> writes:

>On 19/12/2018 15:30, Casper H.S. Dik wrote:


>Aha !
>With the zone halted


>---8<
>root@ytc1-s:~# ls -al /zones/ytcadmin/root/system/volatile/
>total 35
>drwxr-xr-x 2 root sys 3 Feb 28 2013 .
>drwxr-xr-x 5 root root 5 May 2 2018 ..
>-rw------- 1 root root 0 Feb 28 2013 zoneproxy_door
>---8<


>So, I'd guess I have had some sort of bug for over 5 years, but still
>managed to run as an immutable zone ? Anyway I've moved it away, reset
>the file-mac-profile to fixed-configuration, and rebooted.


>It now works.


Hm.

Ah, I see that for some reason we dropped /system/volatile as
a writable path (/usr/lib/brand/solaris/config.xml) for Solaris 11.4.

When an immutable zone does not have writable directory, the kernel
will force that the directory is empty.

The global zone is not immutable?

Many of these file systems for the global zone are now mounted
inside the kernel before the system has started any processes.

The similar is true for zones though in that case the mount is
performed by zoneadmd and that is likely why we changed the
writable files.

Good that your problem is fixed; now that we know what caused
the problem, I'll check why global zones and non-global zones
behave differently.

Casper
YTC#1 (12-20-18, 10:36 AM)
On 20/12/2018 08:05, Casper H.S. Dik wrote:
> YTC#1 <bdp> writes:
> Hm.
> Ah, I see that for some reason we dropped /system/volatile as
> a writable path (/usr/lib/brand/solaris/config.xml) for Solaris 11.4.


Ah, hence why it worked under 11.3

> When an immutable zone does not have writable directory, the kernel
> will force that the directory is empty.
> The global zone is not immutable?


No, in theory the access to it is limited to internal systems. The
immutable zone is ssh accessible from the internet, but fire walled to
prevent onward access.

> Many of these file systems for the global zone are now mounted
> inside the kernel before the system has started any processes.
> The similar is true for zones though in that case the mount is
> performed by zoneadmd and that is likely why we changed the
> writable files.
> Good that your problem is fixed; now that we know what caused
> the problem, I'll check why global zones and non-global zones
> behave differently.


I was a bit surprised that /system/volatile was not created fresh at
each zone boot. Unless I have an old config issue hanging around, I do
tend to get things working and then leave them alone.

However, this zone and my others were migrated from my old AMD server to
this one back in Aug/Oct.
YTC#1 (12-21-18, 11:30 AM)
On 20/12/2018 08:05, Casper H.S. Dik wrote:
[..]
> Good that your problem is fixed; now that we know what caused
> the problem, I'll check why global zones and non-global zones
> behave differently.

Just to add to the weirdness, 1st boot after upgrade and things seem OK.
The previous zone I had to reboot due to changing to PF firewall.

I just rebooted my "next cloud" zone, and hit the same/similar problem
(I'd forgotten that was immutable as well) with root and /tmp ....

Might be worth adding to the bug report for testing purposes.

Zone is fixed-configuration at S11.3
Post S11.4 runs Ok at initial boot
Next boot fails due to file on underlying files on FS mountpoint
I'll start cleaning up ....

---8<
2018-12-21 09:09:44 info: Processing command boot flags 0x1 from
ruid/euid/suid 0/0/0 pid 2700
2018-12-21 09:09:52 notice: NOTICE: Zone booting up
2018-12-21 09:09:52 error: mount of /tmp failed: Permission denied
2018-12-21 09:09:52 error: unable to mount filesystems
2018-12-21 09:09:56 cannot unmount '/zones/ytc-cloud/root': Device busy
2018-12-21 09:09:56 ERROR: unable to unmount /zones/ytc-cloud/root.
2018-12-21 09:09:56 ERROR: Unable to mount zone root dataset.
2018-12-21 09:10:05 notice: NOTICE: Zone boot failed
zone 'ytc-cloud': error: mount of /tmp failed: Permission denied
zone 'ytc-cloud': error: unable to mount filesystems
zone 'ytc-cloud': cannot unmount '/zones/ytc-cloud/root': Device busy
zone 'ytc-cloud': ERROR: unable to unmount /zones/ytc-cloud/root.
zone 'ytc-cloud': ERROR: Unable to mount zone root dataset.
---8<

---8<
root@ytc1-s:~# ls -al /zones/ytc-cloud/root/tmp
total 43
drwxrwxrwt 2 root sys 3 Nov 12 20:36 .
drwxr-xr-x 20 root root 22 Dec 18 17:19 ..
-rw-r--r-- 1 root root 12 Mar 27 2018 sf1h.vjc
---8<
Casper H.S. Dik (12-21-18, 03:13 PM)
YTC#1 <bdp> writes:

>Just to add to the weirdness, 1st boot after upgrade and things seem OK.
>The previous zone I had to reboot due to changing to PF firewall.


That is not strange: the first boot on upgrade will be booted read-only
until it finds that it has done "self-assembly" and then it changes
to immutable state.

On second boot, it is already read-only directly on booth.

>I just rebooted my "next cloud" zone, and hit the same/similar problem
>(I'd forgotten that was immutable as well) with root and /tmp ....


>Might be worth adding to the bug report for testing purposes.


Thanks. I'll file a bug report just before the Christmas break;
it is something we'll need to fix.

Casper
Casper H.S. Dik (01-09-19, 11:10 AM)
Casper H.S. Dik <Casper.Dik> writes:

>YTC#1 <bdp> writes:


>>Just to add to the weirdness, 1st boot after upgrade and things seem OK.
>>The previous zone I had to reboot due to changing to PF firewall.


>That is not strange: the first boot on upgrade will be booted read-only
>until it finds that it has done "self-assembly" and then it changes
>to immutable state.


>On second boot, it is already read-only directly on booth.


>>I just rebooted my "next cloud" zone, and hit the same/similar problem
>>(I'd forgotten that was immutable as well) with root and /tmp ....


>>Might be worth adding to the bug report for testing purposes.


>Thanks. I'll file a bug report just before the Christmas break;
>it is something we'll need to fix.


I did file a bug but did not replied with the bug id, there it
is:

29138660 immutable non-global zones can't boot with turds in /tmp or /system/volatile

I'll fix it shortly.

Casper
YTC#1 (01-09-19, 02:26 PM)
On 09/01/2019 09:10, Casper H.S. Dik wrote:
> Casper H.S. Dik <Casper.Dik> writes:
> I did file a bug but did not replied with the bug id, there it
> is:
> 29138660 immutable non-global zones can't boot with turds in /tmp or /system/volatile


Did you really get it filed with the word "turds" in it ? :-)
Casper H.S. Dik (01-09-19, 04:28 PM)
YTC#1 <bdp> writes:

>On 09/01/2019 09:10, Casper H.S. Dik wrote:


>Did you really get it filed with the word "turds" in it ? :-)


Yes. Seems that our current Solaris repository has no memory of
any other bug synopsis with "turds" in it but our public "userland"
repository also lists a bug with "turds" in the synopsis and so
does our Solaris 10 SCCS history has some such bugs.

There is one "droppings" as well. Not a common word for
bugs, I guess.

It is a technical term.

Casper
YTC#1 (01-09-19, 06:07 PM)
On 09/01/2019 14:28, Casper H.S. Dik wrote:
> YTC#1 <bdp> writes:
> Yes. Seems that our current Solaris repository has no memory of
> any other bug synopsis with "turds" in it but our public "userland"
> repository also lists a bug with "turds" in the synopsis and so
> does our Solaris 10 SCCS history has some such bugs.
> There is one "droppings" as well. Not a common word for
> bugs, I guess.
> It is a technical term.


I used to keep trying to get sh1t in ...... :-)
Similar Threads