experchange > vms

Jairo Alves (01-20-20, 04:19 PM)
Hi,

my VMS box had problemas with DECNETV and after reboot it has
many corrupted permissions.

For example, there are files which cannot be edited by non-privileged users.

When they try to edit, RMS throws those messages:
- "File format is being converted to a supported type"
Then it goes to:
- "Error opening <name-file>; as output"
Then it finihsed at:
- "Insufficient privilege or file protection violation"

But the user has privileges to the file:

(IDENTIFIER=<ID>,ACCESS=READ+WRITE+EXECUTE)

Also, some ACLs are messed up on the entire system disk, like that:

file.txt (APPLICATION,SIZE=%D56,FLAGS=%X0C02,ACCESS=%X00000 087,DATA=%X00000000,%X00000000,%X00000AE0,%X000000 00,%X5CE27E35,%X00000000,%X00000000,%X00000000,
%X00000000,%X00000000,%X00000000,%X00000000)

Any thoughts?
Robert A. Brooks (01-20-20, 04:27 PM)
On 1/20/2020 9:19 AM, Jairo Alves wrote:
> Hi,
> my VMS box had problemas with DECNETV and after reboot it has
> many corrupted permissions.


DECnet (either Phase IV or Phase V) does not corrupt stuff on its own.

Is this node in a cluster with shared storage?

If so, perhaps a node that was not in the cluster inadvertently mounted
for write a disk it should not have?

I hope you have a valid backup of this corrupted system disk.

It looks like RIGHTSLIST.DAT is among the victim files that is corrupted.
Jairo Alves (01-20-20, 04:41 PM)
No cluster, standalone server, rx2660, 8 disks.

System disk is on RAID 1, but that may not help.

> If so, perhaps a node that was not in the cluster inadvertently mounted
> for write a disk it should not have?


That may have been the case. I remember the FAL$SERVER default access was
lost also, after the reboot.

> It looks like RIGHTSLIST.DAT is among the victim files that is corrupted.


I'm not aware of the function of this file, but this is its sec info

Directory DKA1:[SYS0.SYSCOMMON.SYSEXE]

RIGHTSLIST.DAT;1 [SYSTEM] (RWED,RWED,,)
(APPLICATION,SIZE=%D56,FLAGS=%X0C02,ACCESS=%X00000 087,DATA=%X00000000,%X00000000,%X00002AB0,%X000000 00,%X5C8AAA16,
%X00000000,%X00000000,%X00000000,%X00000000,%X0000 0000,%X00000000,%X00000000)
Jairo Alves (01-20-20, 04:46 PM)
Also, if I edit a NEW file, on the same directory, the user can create
the file no problem. However, if we try to edit an existing file, then
those errors are thrown

File format is being converted to a supported type
Error opening <name-file>; as output
Insufficient privilege or file protection violation

I had quotas enabled, and they were giving errors as well,
but I disabled them on the volume to try and see if it was just a quota
issue. It did not help, although the error "quotas exceeded" is gone (obviously)
Arne Vajhøj (01-20-20, 04:49 PM)
On 1/20/2020 9:41 AM, Jairo Alves wrote:
> No cluster, standalone server, rx2660, 8 disks.
> System disk is on RAID 1, but that may not help.
> That may have been the case. I remember the FAL$SERVER default access was
> lost also, after the reboot.
> I'm not aware of the function of this file, but this is its sec info
> Directory DKA1:[SYS0.SYSCOMMON.SYSEXE]
> RIGHTSLIST.DAT;1 [SYSTEM] (RWED,RWED,,)
> (APPLICATION,SIZE=%D56,FLAGS=%X0C02,ACCESS=%X00000 087,DATA=%X00000000,%X00000000,%X00002AB0,%X000000 00,%X5C8AAA16,
> %X00000000,%X00000000,%X00000000,%X00000000,%X0000 0000,%X00000000,%X00000000)


I have not seen such ACL's since I last played with Pathworks like
20 years ago.

You have not by any chance mounted disks for Pathworks usage and let PC
users edit files?

Arne
Stephen Hoffman (01-20-20, 06:00 PM)
On 2020-01-20 14:19:59 +0000, Jairo Alves said:

> my VMS box had problemas with DECNETV and after reboot it has many
> corrupted permissions.


It's possible that something stomped on the storage device, but there
are lots of possibilities there.

And as per my usual. I'd recommend an upgrade from Phase V to Phase IV,
unless you specifically need a feature such as DECnet over IP. And
y'all seriously want to get off of DECnet, as it's utterly insecure.

> For example, there are files which cannot be edited by non-privileged users.
> When they try to edit, RMS throws those messages:
> - "File format is being converted to a supported type"
> Then it goes to:
> - "Error opening <name-file>; as output"
> Then it finihsed at:
> - "Insufficient privilege or file protection violation"


File format conversions are common when using somewhat mismatched
tools, and the editor switches to the unspecified-editor-expected
format, even if that format is wrong for subsequent use by the original
application.

The error opening for output can be detected with security auditing or
security alarms, but usually involves the journal file used by editors.
That directory can often be redirected with a logical name specific to
the particular editor.

This whole editor journaling approach works, but predates versioning
schemes such as those used for file recovery and rollbacks and backups
that's available within macOS and other platforms.

> But the user has privileges to the file:
> (IDENTIFIER=<ID>,ACCESS=READ+WRITE+EXECUTE)


That's probably better thought of as "rights" and not "privileges" as
that doesn't involve privileges.

If you're using apps to access the data, SUBSYSTEM identifiers are
quite handy as those can be granted transiently while the app is
running, and not via static user mappings within RIGHTSLIST.

That or a similar ACE will usually also have to be applied to the
encompassing directory, and there'll usually need to be DEFAULT ACEs
added to allow new files to be accessible with the expected access
permissions.

I'll assume most or all of that's all in place...

> Also, some ACLs are messed up on the entire system disk, like that:
> file.txt
> (APPLICATION,SIZE=%D56,FLAGS=%X0C02,ACCESS=%X00000 087,DATA=%X00000000,%X00000000,%X00000AE0,%X000000 00,%X5CE27E35,%X00000000,%X00000000,%X00000000,
> %X00000000,%X00000000,%X00000000,%X00000000)


That's an APPLICATION ACE, and mostly harmless, and that and other
APPLICATION ACEs do not mediate any object access nor any file access.
That's used by PATHWORKS and various other products as a metadata
store. There was a third-party OpenVMS Windows file transfer tool that
used these ACEs some years ago, too. May still be. Could also indicate
a file header or extension header corruption. The ACL is stored in the
file headers associated with the file, and not in the RIGHTSLIST. The
identifier ownership mapping is stored in RIGHTSLIST.

> Any thoughts?


OpenVMS I64 version on this rx2660? V8.4 prior to UPDATE V5.0 had a
storage corruptor. I got hit with that corruption, as did some other
folks. You really don't want to be on an un-updated or low-updated
OpenVMS I64 V8.4, for instance.

Also check the firmware revision on the RAID controller, etc.

You're probably headed for rolling in backups from prior to the DECnet
Phase V issues, assuming those exist—if this is a storage corruption.
I don't yet see specific evidence of that corruption here, though. All
of what's described could be normal for some system configurations. Or
yes, if this behavior and this ACE is all new and this all worked and
this wasn't visible prior to DECnet Phase V going weird, then this
certainly could be evidence of a corruption. If the file headers have
been corrupted, then this volume is best only used as a source of data
for a newly-initialized volume, merging the data with the available
backups.
Simon Clubley (01-20-20, 08:45 PM)
On 2020-01-20, Jairo Alves <jairo.ptbr> wrote:
> Hi,
> my VMS box had problemas with DECNETV and after reboot it has
> many corrupted permissions.
> For example, there are files which cannot be edited by non-privileged users.
> When they try to edit, RMS throws those messages:
> - "File format is being converted to a supported type"


These are sequential files I assume ? Ie: somebody isn't trying to edit
an indexed file in an editor ?

Things to try:

dir/full on RIGHTSLIST.DAT. Do not use a version specification. Look to
see if there's more than one version and when the versions were created.

If dir/full reveals the file format of the latest version is sequential,
then you have found the problem. If it's indexed, make a copy somewhere
of the latest version outside of the system directories while the system
is idle and then do an analyze/rms on the copy.

If that completes without error, then use authorize to look at the
RIGHTSLIST.DAT entries and make sure they are as expected.

Is "show error" reporting any unexpected disk errors ?

Have you considered running analyze/disk on the disk ? Please read the
manual and the online help before you consider doing this.
WARNING: Do _NOT_ try to repair anything unless a known good backup is
available.

Be aware that analyze/disk in read-only mode on a live system will
likely report random errors as the disk contents change. You are looking
for the errors that remain the same during multiple runs.

Simon.
Jairo Alves (01-20-20, 10:40 PM)
> You have not by any chance mounted disks for Pathworks usage and let PC
> users edit files?
> Arne


I tried to restart DECNET via the (desperate) command stop/network decnetv
this made the box go dark. So, I had to shut it down via ILO (CM/PC-off)

Then, it didnt mount everything nicely on the first try.

I then rebooted it again and things "seemed" normal. But they aren't.
Jairo Alves (01-20-20, 11:08 PM)
> And as per my usual. I'd recommend an upgrade from Phase V to Phase IV,
> unless you specifically need a feature such as DECnet over IP. And
> y'all seriously want to get off of DECnet, as it's utterly insecure.


We work in Industrial Automation, behind DMZ and disconnected from the web.

A lot of stuff depends on DECNET, necessarily Phase V, since behind a DMZ.

> File format conversions are common when using somewhat mismatched
> tools, and the editor switches to the unspecified-editor-expected
> format, even if that format is wrong for subsequent use by the original
> application.


I'm trying to edit with "edit" editor ($ ed a.csv)

> The error opening for output can be detected with security auditing or
> security alarms, but usually involves the journal file used by editors.
> That directory can often be redirected with a logical name specific to
> the particular editor.


If journalling, would this be compatible with the user having no problem
editing a brand new file or even the same file on other disk?

> This whole editor journaling approach works, but predates versioning
> schemes such as those used for file recovery and rollbacks and backups
> that's available within macOS and other platforms.


> That's probably better thought of as "rights" and not "privileges" as
> that doesn't involve privileges.


Yes, thanks for the heads up. That being said, I think the RMS error
generator sometimes will report lack of privileges to the file when
the user doesn't have permission.

> If you're using apps to access the data, SUBSYSTEM identifiers are
> quite handy as those can be granted transiently while the app is
> running, and not via static user mappings within RIGHTSLIST.


Could you explain this part again? The use case is a DCl script creating
and writing to a CSV file.

> That or a similar ACE will usually also have to be applied to the
> encompassing directory, and there'll usually need to be DEFAULT ACEs
> added to allow new files to be accessible with the expected access
> permissions.


Yes, the directory has the ACEs. See:

Directory D$VXLSUP:[000000]

byp_ov.DIR;1 [VXLSUP] (RWE,RWE,RE,E)
(IDENTIFIER=BYPMAN,ACCESS=READ+WRITE+EXECUTE)
(IDENTIFIER=BYPMAN,OPTIONS=DEFAULT,ACCESS=READ+WRI TE+EXECUTE)

> I'll assume most or all of that's all in place...


> file headers associated with the file, and not in the RIGHTSLIST. The
> identifier ownership mapping is stored in RIGHTSLIST.


Should I have a backup of the Rightslist file?

> OpenVMS I64 version on this rx2660? V8.4 prior to UPDATE V5.0 had a
> storage corruptor. I got hit with that corruption, as did some other
> folks. You really don't want to be on an un-updated or low-updated
> OpenVMS I64 V8.4, for instance.


$ pipe prod sho hist | sea sys$input openvms
HP I64VMS OPENVMS V8.4 Platform Install Sys 14-MAR-2013

pipe prod sho hist | sea sys$input upd
HP I64VMS VMS84I_UPDATE V6.0 Patch Install Val 14-MAR-2013

I guess I'm on update 6, probably.

> yes, if this behavior and this ACE is all new and this all worked and
> this wasn't visible prior to DECnet Phase V going weird, then this
> certainly could be evidence of a corruption. If the file headers have
> been corrupted, then this volume is best only used as a source of data
> for a newly-initialized volume, merging the data with the available
> backups.


I'm affraid those ACEs are new and also the FAL$SERVER stopping working
is not normal at all. I think it may have lost also the TCPIP proxies,
but since I don't have an easy backup of those, I can't be certain.
johnwallace4 (01-20-20, 11:51 PM)
On Monday, 20 January 2020 14:20:01 UTC, Jairo Alves wrote:
[..]
> file.txt (APPLICATION,SIZE=%D56,FLAGS=%X0C02,ACCESS=%X00000 087,DATA=%X00000000,%X00000000,%X00000AE0,%X000000 00,%X5CE27E35,%X00000000,%X00000000,%X00000000,
> %X00000000,%X00000000,%X00000000,%X00000000)
> Any thoughts?


What leads you to think that DECNETV was the *cause* of these problems?

Is there any *evidence* that DECNETV was anything other than an innocent
victim of problems elsewhere on the system (or maybe its storage)?

Does the system in question have an operator log, an accounting log, an
Itanic variant of what used to be the device error log, the usual
security logs, etc? Depending on the site-specific configuration and
procedures, there may also be relevant FAL logs.

Have you used any of the VMS-provided utilities to see what they make of
the allegedly corrupt files, or to check the consistency of filesystems,
etc (in a safe, read-only, fashion if possible)?

Have there been any error messages that were more helpful
than e.g.
"it didnt mount everything nicely on the first try." ?

The inability to edit files in the way you've used in the past is a
symptom, but not a particularly informative one.

Do you have access to professional VMS support, either onsite or via
secure remote access?

If this system contains anything of value, then based on the
description so far, I'd want to be sure that leaving the system
up and running isn't going to cause further data loss (or other
longer term issues). So far, it looks like you may be at risk.
But you're closer to the setup than us armchair experts are.

If this is an industrial automation box associated with the VXL
package or similar, is the vendor/provider able to help at all?
(Or is that your role??????)

Best of luck.
Stephen Hoffman (01-21-20, 02:11 AM)
On 2020-01-20 21:08:20 +0000, Jairo Alves said:

>> Hoff: And as per my usual. I'd recommend an upgrade from Phase V to
>> Phase IV, unless you specifically need a feature such as DECnet over
>> IP. And y'all seriously want to get off of DECnet, as it's utterly
>> insecure.

> We work in Industrial Automation, behind DMZ and disconnected from the web.


Saw a recent study on air-gapped networks that weren't. Yours likely
isn't intended to be air-gapped, which means bypassing that firewall is
easier.

> A lot of stuff depends on DECNET, necessarily Phase V, since behind a DMZ.


A lot of stuff that hasn't been updated or overhauled in most of thirty
years or so, and probably won't be, until and unless a breach. Or a
wholesale replacement. If past practices elsewhere are any guide.

DECnet is still insecure.

SCADA and industrial automation is quite familiar, and it's an
ever-increasingly-valuable target, too. Did a whole lot of
DECnet-based work and OpenVMS work in similar factory floor
environments. The DECnet factory floor work was ~30 years ago, and
effectively wide open. Nobody'd even consider that approach now,
absent specific compatibility requirements.

>> Hoff: File format conversions are common when using somewhat mismatched
>> tools, and the editor switches to the unspecified-editor-expected
>> format, even if that format is wrong for subsequent use by the original
>> application.

> I'm trying to edit with "edit" editor ($ ed a.csv)


Depending on the OpenVMS version, that might be EDT or EVE. Probably
EVE, but folks can and do reset access to EDT using a DCL symbol.
Maybe ED or ED*IT, in this case.

>> Hoff: The error opening for output can be detected with security
>> auditing or security alarms, but usually involves the journal file
>> used by editors. That directory can often be redirected with a logical
>> name specific to the particular editor.

> If journalling, would this be compatible with the user having no
> problem editing a brand new file or even the same file on other disk?


Depends on local settings.

Use security auditing or alarms or (for this case) SET WATCH and see
what's being accessed, and what's being blocked, and why.

> Yes, thanks for the heads up. That being said, I think the RMS error
> generator sometimes will report lack of privileges to the file when
> the user doesn't have permission.


That's because the privileges are the last things checked, and also
because OpenVMS typically can't markedly alter old and potentially
confusing status codes. Because reasons.

>> Hoff: If you're using apps to access the data, SUBSYSTEM identifiers
>> are quite handy as those can be granted transiently while the app is
>> running, and not via static user mappings within RIGHTSLIST.

> Could you explain this part again? The use case is a DCl script
> creating and writing to a CSV file.


SUBSYSTEM identifiers cannot be associated with DCL procedures.

There are some open-source packages that allow running procedures with
privileges, but OpenVMS itself doesn't.

For more information on SUBSYSTEM identifiers, see the OpenVMS security manual.

> Should I have a backup of the Rightslist file?


Only of a RIGHTSLIST that you want to keep.

If this system is often or always live, CONVERT /SHARE is one potential
work-around for a mostly-consistent copy.

> $ pipe prod sho hist | sea sys$input openvms
> HP I64VMS OPENVMS V8.4 Platform Install Sys 14-MAR-2013
> pipe prod sho hist | sea sys$input upd
> HP I64VMS VMS84I_UPDATE V6.0 Patch Install Val 14-MAR-2013
> I guess I'm on update 6, probably.


V8.4 UPDATE V6.0 does have the file system corruption fix. You're well
behind current for HPE, and further for VSI, though.

> I'm affraid those ACEs are new and also the FAL$SERVER stopping working
> is not normal at all. I think it may have lost also the TCPIP proxies,
> but since I don't have an easy backup of those, I can't be certain.


You could well have been hacked, too. But again, APPLICATION ACEs do
not alter file access processing.
Jairo Alves (01-21-20, 02:34 PM)
> What leads you to think that DECNETV was the *cause* of these problems?

Nothing, it may as well be a symptom of some disk problem. The fact is
DECNET stopped working and it is essential, so I had to do something.

As I said, I tried a "stop network DECNET" and it stopped everything
network related, so I lost SSH to the machine as well.

I suspect the fatal error probably was, then, shutting it down
via ILO PC-OFF. But this server sits on a Datacenter and it would
be troublesome to go physically there to restart the network.

In restrospect I would prefer to go there instead.

> Does the system in question have an operator log, an accounting log, an
> Itanic variant of what used to be the device error log, the usual
> security logs, etc? Depending on the site-specific configuration and
> procedures, there may also be relevant FAL logs.


Operator.log has now only two messages that are unusual, but I can't
understand what they are.

%%%%%%%%%%% OPCOM 21-JAN-2020 04:17:38.92 %%%%%%%%%%%
Message from user SYSTEM on I16000
Event: Deleted Remote NSAP from: Node LOCAL:.I16000 OSI Transport Local NSAP IP_ANY,
at: 2020-01-21-04:17:38.913-03:00Iinf
Name=0AAA710B,
NSAP Address=,
UID=F5B9587E-3C04-11EA-AA0E-001CC4FC41BB,
Creation Time=2020-01-21-04:17:38.832-03:00Iinf,
Remote Protocol Errors=0,
Local Protocol Errors=0,
Total Octets Received=0,
Total Octets Sent=0,
PDUs Received=0,
PDUs Sent=0,
Duplicate PDUs Received=0,
Retransmitted PDUs=0,
Connects Received=0,
Connects Sent=0,
Rejects Received=0,
Rejects Sent=0,
User PDUs Discarded=0,
User Octets Received=0,
User Octets Sent=0,

%%%%%%%%%%% OPCOM 21-JAN-2020 04:17:38.92 %%%%%%%%%%%
Message from user SYSTEM on I16000
User PDUs Received=0,
User PDUs Sent=0,
Congested PDUs Received=0
eventUid F5C5B48E-3C04-11EA-AA0E-001CC4FC41BB
entityUid BEA2EFFB-313C-11EA-81CA-001CC4FC41BB
streamUid 9AA6C180-313C-11EA-88B2-AA000400173C

FAL Logs? I don't know about them, unfortunately. After FAL$SERVER stopped
working, I simply created explicit TCPIP Proxies. I need to provide this
access, so what else to do.

> Have you used any of the VMS-provided utilities to see what they make of
> the allegedly corrupt files, or to check the consistency of filesystems,
> etc (in a safe, read-only, fashion if possible)?


No, would you suggest a specific route?
I suspect some error mounting the disks corrupted the permission info.
Also, no file contents were harmed, as far as I could see.
And also, only the SYSTEM disk was affected, none of the other 7 disks.

> Have there been any error messages that were more helpful
> than e.g.
> "it didnt mount everything nicely on the first try." ?


Not that I have seen. Just problems mounting the disks on the first reboot
(note that this machine was more than 2 years without rebooting, it's a
server that does not reboot regularly, only once every several years)

> Do you have access to professional VMS support, either onsite or via
> secure remote access?


Yes, but they are HPE Openvms support, they barely have a single person
in Brazilian openvms support.. Usually they think we run Unix on the rx2660,
It has been totally useless to be honest.
Hopefully we will transition to VSI support in the near future..
The US support should be infinitely better.

> If this system contains anything of value, then based on the
> description so far, I'd want to be sure that leaving the system
> up and running isn't going to cause further data loss (or other
> longer term issues). So far, it looks like you may be at risk.
> But you're closer to the setup than us armchair experts are.


Further data loss I expect not.

> If this is an industrial automation box associated with the VXL
> package or similar, is the vendor/provider able to help at all?
> (Or is that your role??????)


Yes, this machine supports as a central hub to the hundred+ VXL rx2660s
we have in operation.
Jairo Alves (01-21-20, 03:21 PM)
> DECnet is still insecure.

We know, but legacy systems are difficult things to change, especially
when change means a lot of capex for no other benefit other than security.

> SCADA and industrial automation is quite familiar, and it's an
> ever-increasingly-valuable target, too. Did a whole lot of
> DECnet-based work and OpenVMS work in similar factory floor
> environments. The DECnet factory floor work was ~30 years ago, and
> effectively wide open. Nobody'd even consider that approach now,
> absent specific compatibility requirements.


Issue is: VXL (SCADA software for VMS) is based on VXLnet, that is 100%
dependent on DECNET. So, to give up DECNET is similar in investment to
give up everything from the OS to the SCADA software. Very difficult to do.
Simon Clubley (01-21-20, 03:28 PM)
On 2020-01-21, Jairo Alves <jairo.ptbr> wrote:
> No, would you suggest a specific route?


Did you try any of the things I suggested ?

Simon.
Jairo Alves (01-21-20, 07:33 PM)
> Did you try any of the things I suggested ?

"VMS-provided utilities to see what they make of.."

I don't know them, I tried this:

set file /attr=(RFM:VAR) my-file.csv

Didn't help.

Similar Threads