ast: segfault with extended globs

@krader1961, thoughtless phrases commonly used to get rid of “pesky” people do not change anything. IMHO actually not @jghub but you should go away.

Well, some years ago I decided to only waste minimal time in this repo: just skim its archive from time to time, add some thumbs if it is really worth, and warn people, that this repo got hijacked if needed. Reasons are obvious. But since @jghub asked, and he really deserves to know, that he is not alone because of his admirable tenacity/power to get things into the right direction (I really appreciate it), and I got tired to explain the situation to people again and again, here my 10¢. If TLDR, just scroll down to the end.

Fact is, the repository got hijacked and is in bad state:

Once upon a time, there was a Advanced Software Technology (AST) department AT&T, where David Korn (dgk), Glenn Fowler (gsf), Kiem-Phong Vo (kpv) and others developed a lot of very smart stuff like ksh93. Around 1999 AT&T changed its mind and allowed AST to make it public. Since than features/patches etc. were discussed in personal mails or AT&T mailing lists, and occasionally gsf packaged the sources and published it on the AST web site (IIRC since 2000).

After dgk and gsf job’s at AT&T got terminated in Oct 2013 (kpv already left some month ago)[1], dr.ek “took over” AST site maintenance and synced in gsf’s packages like ast-open-2014-12-24.tgz [2], what many people erroneously took as a ksh stable release (ksh93v-) - note the trailing dash, i.e. “beta”. This version is known to have so many bugs, that it “should not deemed release-worthy” [3]. Furthermore the lack of a public issue tracker and source code repository was mentioned as a major hindrance (Sep 2015).

At the end of 2015 the AST website went off the air (official statement: security issues) and finally dr.ek was so nice to push the AST leftovers to github in Jan 2016 [4]. Unfortunately after this, it became obvious that neither dgk/gsf/kpv/dr.ek nor anyone else at AT&T had/has time to devote to the repo.

So once dannyweldon contacted dr.ek (lkoutsofios) to ask about some things a while ago, he offered him commit access. He accepted and started to sort/tag issues submitted so far (~ Mar 2017). However, Danny’s time was/is limited as well, and when siteshwar announced his plans to fork the AST repo, Danny thought some thing like “Oh nice that RedHat takes over a let a professional work full time at it” and suggested to ask dr.ek to give him commit access to the AST repo not knowing, that he is in bad company (i.e. Kurtis Rader (krader)). Finally he got it [5] and krader got access obviously from him ~ Nov 2017, possibly being still blended by his boring “I’ve worked for …” in the fish project. They started immediately damaging the repo as well as ksh93 itself. Seeing this, Danny started to regret this accident wholeheartedly (and I guess he still does).

So both started to effectively kill most of the software of the AST repo, even so many people asked them to not do that but to fork and than start playing around. They even ignored the original owners of the repo[6]. In krader’s case I guess, as usual he did not understood, what dr.ek is saying, because he rarely does its homework (i.e. carefully study the source, its background/context and related material) and therefore ignored it. In this special case krader did and still does not even know (and obviously siteshwar as well), that it was damn easy to build just the ksh93, that there is no need at all to build the whole AST tree.

Anyway, what I see is, that especially krader continues to not listen to people (only what he thinks rules), is misinformed about the software he tinkers with, ignores things he does not understand, obviously does not try to study and fully understand the required material (e.g. AST malloc) and prefers to rant about “how stupid and confusing the existing code is” [7]. Well, he actually does not even have a ksh93 background and continues to disparage ksh93 as a batch facility for his CLI calls [8][9], to kill very essential stuff, he does not understand [10][11][12] or he does not like [13][14], or to mess it up/replace it with other crap he has seen in other software [15]. One could think, his target is to kill ksh, to throw away all features, which actually make it different and so useful for real ksh users. Anyway, many people seem to have stopped wasting their time in such an user/developer unfriendly environment [16]. I’m still wondering, why krader tries to shoutdown others by hammering his working places into discussions all the time. In what dungeon have they put him that he did not realize, that there are other smart people working at, or got job offers from those companies as well and that there are also a lot of smart people work at small companies, universities, etc., there where most innovation actually happens? Anyway, in 2017 dgk mentioned, that he is still interested in ksh93 and wants to get back, when he gets some time for it (PM). I guess, he got scared off as well and now devotes his time to more valuable things.

And BTW, beside trying to remove very important, unique, very usable features of ksh93, he has shown several times, that he is completely unqualified as a maintainer of this [or any other] software. Insulting user which withstand his misinformation and lame argumentation, declaring builds as being a stable release, when it is known, that this is definitely not the case, closing issues from people he does not like (because they are right, or do not worship him), or removing important code just because he does not like the OS (and/or its vendor) using it [17], is not what I expect from a maintainer (some people already named his behavior that of an uneducated stubborn child and behaves like an elephant in the china shop[18]).

So yes, at least for me all this is exactly, what hijacking involves:

• got access by pretending to do something different as they actually have done
• neither krader nor siteshwar have any legitimization to call themselves “maintainer” of ksh93. Or did I miss any community poll or election or whatever regarding this?
• Instead of being honest and fork, they still hide what they produce under the coat of AT&T/AST and make people walk into this trap (as Debian did)
• most of the people trying to contribute/sail on the ksh93 ship got scared off especially by kraders rude, violent behavior and successfully killed the ksh93 community [19]
• It appears that at least krader is absolutely incapable to maintain software, especially this/a language (he seems to understand C and tooling around it pretty good, but almost nothing at a higher level like ksh93 features) and has no desire at all, to get it to its intended destination.
• krader seems to be very narrow-minded and has no desire to let ksh93 evolve or to enhance it with new features or to document, what he thinks is undocumented (instead he prefers to remove it).
• He doesn’t care about breaking it, i.e. making it totally unusable for its users and existing ksh93 scripts.
• Last but not least, setting up a closed mailing list, so that only he and his believers can communicate, clearly shows krader’s intention to keep out ksh93 community members, users, developers and keep this repo hijacked.

IMHO, this repo should be restored to its original state in Jan 2016, a clone be created as dr.ek suggested (/att/ksh93). Whether the master branch of it gets deduced from u+ or v- might be discussed there (I prefer u+ and migrate step by step changes from v- to it in a separate branch). I can live with siteshwar as a maintainer since he seems to have proper guidance and seems to loose his blindness wrt. krader, but krader itself should do, what he suggested several times to other people: he should create his own fork and if he likes submit his changes as normal Pull Requests (PR). IMHO he should never get commit access to the new ksh93 repo again (and neither to the AST repo too of course).

While I had mainly listened and monitored this repository in the past only, I also cannot stand the current situation anymore and would like to express my full support for @jelmd and @jghub and their idea to revoke direct commit access for @krader1961 to this att/ast repository.

While at the starting point of kraders’ introduction to this repository I was still quite optimistic and found some of the changes he introduced also - at least - partly welcome, it now (after all these years) ended up in a complete mess which not only damages the legacies of David Korn, but also imposes some severe risk that the ksh shell (which has some clear advantages for non-interactive shell scripting) will not survive this if this situation is not resolved soon and the official ksh/ast repository not returned into the hands of the ksh community again.

Not only has @krader1961 introduced IMHO a quite uncommon build environment (meson+ninja) for such a legacy unix project (which introduces some unnecessary dependencies, e.g. python) and should have better used autoconf or cmake for such a C-only project which should be IMHO easily compilable on all different kinds of unix platforms from legacy platforms like solaris, hpux over complex x86 based linux system till resources limited environments like embedded systems. Furthermore, he introduced breaking changes or performance degradations without discussing these changes in advance in the ksh community and he is constantly ignoring concerns from other ksh community members that have a much broader knowledge about the scripting language or the functionality which makes ksh special.

But what I found most irritating is, that @krader1961 obviously severely lacks to have competences to act as a proper open source maintainer for such an important project where communication skills are absolutely necessary to bring together all different opinions and concerns. IMHO this project lacks a clear code of conduct which is common for all modern open source projects (which krader would have already violated multiple times). Furthermore, he is IMHO also missusing the AT&T brandmark by directly committing breaking changes to this branded repository rather than having forked it under his own account name and submitting all changes to that fork instead and then sending in pull requests to the original att/ast repository which someone else (someone from the original “att” group) would have to approve after having discussed this carefully in the ksh community. By directly committing to this repository external viewers might get the false impression that these changes are offcially approved by AT&T or the ksh community – which in fact both is not the case!

Thus, I would like to again express my deep support for the idea to revoke direct commit access for @krader1961 and restore this repository to the original ksh93u+ state. Then krader can fork this repository, integrate his changes into his own fork, rename it “kradersh” and perhaps submit each and every of his suggested pieces to the original att/ast repository via official pull requests which external reviewers can then check and integrate into a dedicated development branch after having discussed them carefully like this should have been done right from the start. And least not least, a detailed code of conduct should be added to the repository which defines under which conditions contributions and discusssions have to be done. Looking back, this would have probably helped to prevent this mess right from the start…

Hi all!

I am not knowledgeable about ksh but I have plenty of open source experience, and I am an admin on this repo since I work at AT&T. I’m going to help @lkoutsofios with this transition, however it goes.

It’s a tough situation when the original maintainers abandon a project with no successors, and especially tough when they abandon it involuntarily.

It sounds like we are close to a consensus that the repo should be backed out to a prior version, and at least @krader1961 should fork if he wants to take it in a different direction. But there are a few conflicting proposals about what version this repo should be backed out to.

There shouldn’t be any bad feelings in forking a repo when the original maintainers have left. It’s just a shame that there wasn’t any governance in place and we ended up with multiple maintainers with commit access but different goals.

Again, I’m not a stakeholder or knowledgeable about the technical issues, but in some ways backing out to a truly stable early version sounds cleanest to me. Cleanest in the sense that this fork would be the “AT&T Version”. Then others can fork whatever other versions they want.

If I back anything out, I will of course leave other versions on branches. Nothing will be destroyed.

I’m open to all suggestions. I will follow the discussion and see if there are other stakeholders and other opinions.

(I am not going to become a long-term maintainer; just want to help resolve this conflict and restore the repo with my git skills.)

@gordonwoodhull @johrstrom @kindofblue @MarshM @s-u @tremaineeto

dear owners of the att/ast repository,

having been encouraged to do so by some followers of this project and after having spend a couple of months trying to provide constructive feedback completely unsuccessfully I’d like to bring to your attention the sorry state of this project.

I would appreciate if you would take the time and closely look at what is happening to this repository and to assess what damage is done to the reputation of D. Korn and ATT by kurtis rader (krader1961) who effectively has taken over command in this project and is severely misbehaving while also harming the code base.

please read, e.g., through the full history of the present issue and look at the full #1449 issue or this conclusion https://github.com/att/ast/issues/1449#issuecomment-581041219

there are also multiple instances of diverse people objecting against what is happening here over the full history of the project if you go back to the history of this project. #88 is just one early example for the complete misguidedness and ignorance of krader

a few further examples of the inadequacy of krader’s personality as the de-facto lead (or even a contributor) to this project:

“Gah! Don’t get me started about the issues in the nameref code. It’s a steaming pile of caca.” see here: https://gitter.im/ksh93/users

“> P.S., I am deliberately not including the mailing list address in this reply. Same as my earlier, private, replies in order to avoid making disagreements that are personality based public. That you are deliberately making them public tells me you are an asshole.” see here:

https://groups.google.com/d/msg/korn-shell/TVpHd5rrI5g/8Cj6ir7NAwAJ

someone acting in this way should no longer have commit access to this repository in my view.

Bug reports are great and welcomed. But given your intense interest in a project you’ve repeatedly claimed has been “hijacked” I’d be more impressed if you actually provided an analysis of the bug and, ideally, a fix. If nothing else you could have provided a backtrace of the failure.

fake fact fighting cont’d:

• I don’t have an “intense interest” in this project. I do have interest in availability of a fully functional ksh93 – and ksh2020 is becoming something different and inferior in your hands.

• it was someone else who described what has happened to the ast project as “hijack”. you might check this and also whether I’ve adopted and used that term at all afterwards (I don’t think so). “hijack” is close but I’d rather describe the situation as a “hostile takeover” since the ksh community very obvious and nearly unanimously does not agree with your actions. and “hostile” is a good summary of your overall attitude anyway.

concerning the rest of your comment: nobody wants to impress you. nobody is impressed by you. you received a bug report. react any which inappropriate way you want.

I do not think that downstream consumers care too much whether they download release tarballs of ksh from att/ast or ksh-reloaded/ksh, as long as the software works well for them. For Red Hat it means basically two things:

1. Scripts written for ksh93 continue to work. This is not much compatible with the approach of @krader1961.
2. We are able to maintain it (build the software in predictable way and in reasonable time, we are able to fix bugs and security issues, etc.). This is not much compatible with the goals of some community members (requesting to keep support for decommissioned HW/SW, etc.).

If AT&T and the original community want to keep the brand solely for ksh93u+, it makes sense to me. We can rollback the master branch (while keeping ksh2020 in a non-default branch to not loose all the info recorded in issues and pull requests) and put there a README clearly stating that ksh93u+ is not being developed any more and list the existing forks with their pros and cons.

Would it make sense?

@jelmd: thanks a lot for speaking up! it seems high time to bring this bad development to and end. also many thanks for providing a lot of historic detailed background information I did not have – although I had put the main pieces together already.

and further thanks for compiling additional extreme horrific examples of what krader has done to ksh (“kill non-forking subshells!?” that maybe so far is the craziest idea/deed(?) of them all).

I also would opt for restarting from u+ rather than v- for said reasons (but that is only an opinion based on personal experience with instability of v- and what people have said, not from a deeper analysis, so …) and to start backporting fixes.

I have talked to a colleague: if these necessary steps are not taken here (depends on the reaction of the ATT people and siteshwar – or absence thereof), we are determined to try to wrap 93u+ in a new build system (gmake, Cmake are candidates) and see our way from there. if that becomes necessary, maybe we can join forces regarding integration of patch sets (solaris, eg.?) etc.?

Also, if you’re using ksh for this type of problem you are using the wrong tool for the job. There are problems where ksh is the best tool. This isn’t one of them. Google “if all you have is a hammer.”

@kdudka: regarding how exactly to make the transition

It does not really matter as long as the references to the issues and pull requests (and the references inside them) stay valid.

after changing the build system (and nothing else!) that would provide the legacy ATT ksh93u+ shell for distros to use

The problem is that this task is more difficult than one would expect. If you want to keep support for all the obscure HW and SW at the same time (as repeatedly requested here), it is nearly impossible.

and further bug fixes could than happen on master. but with a policy similar to the one observed by the AST team of D. Korn et al.: carefully and one by one and with a very conservative attitude of integrating patches/pull request after thorough discussion and testing

I think you are underestimating the amount of work needed for this. Most of the users report just issues without patches. Someone has to debug them, write the patches, and review the patches. Given the shape that ksh93u+ is currently in, I think it requires at least one full-time employee dedicated to this.

if this is of interest: @jens-maus has signalled principal willingness to look into the build system issue (considering Cmake so far as the way to go). I am confident he would do a good job. I would be able to help in testing (not so much with the Cmake stuff). any opinions whether this would be welcome?

This would certainly be welcome and I wish you good luck with that!

principally, I agree with everything @gordonwoodhull has proposed/decided: thank you.

I disagree with @siteshwar that this would be the “death of ksh”. rather to the contrary. going on without a clear cut would do decisive further (maybe mortal) harm to ksh. and the ‘visibility’ he talks about seems to be mostly spurious (or identical with the visibility here where there is near unanimous agreement that ksh2020 went off on a bad tangent right from starting with ksh93v-). maybe @siteshwar can reconcider his position to prevent fragmentation?

I also second @kdudka that downstream users are just interested in a predictably working true ksh93(u+…, for the time being) and the consequences this has for distros (not only RH). except that I am not sure (and have not seen statements to that extent around here that I can recall) that support of ancient HW/SW (beyond strict backward compatibility of ksh language-wise) is a crucial issue for a sizeable part of the community. so maybe there is no collision of interests regarding your second point at all, making it a non-issue?

@kdudka: regarding how exactly to make the transition, I am not sure whether I completely understand your proposal/intention. but the main idea seems to keep the present repo (instead of starting over from “clean” repo) and restructure it? maybe that would be good, but depends on what exactly is the plan.

I would like the idea of putting ksh2020 on a “dead end” branch (if that is the plan) to maintain easy access to the issues (however questionable their factual correctness, partly…) and fixes that are relevant for 93u+ (possibly not so many since most fixes related to ksh93v- and ksh2020 only problems…) . but by no means should that branch see then further activity in my view: ksh2020 development (= krader) needs to happen in a fork and the ksh2020 branch in the present repo should not accept pull requests but be in a “sealed/archived” state just for reference. development would exclusively happen for the ksh93u+ line. I would find this clear separation of ksh2020 and ksh93 mandatory for all the reasons that have let to the present need of a reset and in order to definitely avoid replication of the confusion out there between ksh93 proper and ksh2020. the distinction needs to be very clear.

it then seems now consensus (except with @siteshwar, possibly) that master should be rolled back to prestine 93u+. this would be good. after changing the build system (and nothing else!) that would provide the legacy ATT ksh93u+ shell for distros to use (modulo first including the known solid patches from different source but without overdoing it at this point?).

if that state is reached the many year hiatus in easy ksh93u+ availability would be at an end (which is the only real danger for ksh’s survival in my view). that would probably be the time to put the result on a “ksh93u±legacy” branch.

and further bug fixes could than happen on master. but with a policy similar to the one observed by the AST team of D. Korn et al.: carefully and one by one and with a very conservative attitude of integrating patches/pull request after thorough discussion and testing (in order to avoid duplication of the ksh2020 mess with a multitude of cuts and behavioural and feature changes/deletes nobody ever kept track of or cared to document beyond commit comments of questionable quality and reliability).

if this is of interest: @jens-maus has signalled principal willingness to look into the build system issue (considering Cmake so far as the way to go). I am confident he would do a good job. I would be able to help in testing (not so much with the Cmake stuff). any opinions whether this would be welcome? or whether to categorically stay with autoconf? or even meson (but most people do not seem to like that road)?

The authentic AT&T Version will be ksh93u+.

Since there is no one inside the company to maintain the project, that will be the end of the original fork, and we can archive it.

I’ll be glad to help port issues to another repo, if desired, and update the README to point to any forks which continue from that point.

If there is any volunteer to maintain a community fork, they could use their own account, or start e.g. a ksh organization. (A common solution in this situation.)

@gordonwoodhull: I am very happy that my “SOS call” reached you (it was a last resort after months of trying otherwise) and that you care to intervene/help.

a first feedback: my best guess is that @siteshwar will have a somewhat different view from the rest of the followers of this project regarding how/whether to backout and start over. that is, I understand he does not want to do that but to go forward from the current state of the project – which I would find problematic at the very least.

so I would agree completely with what you seem to propose: go back to last truly stable version (that most definitely is 93u+, impossible it could be 93v-) and make it ‘longtime survivable’ (new build system) as “the ATT ksh93” shell. what is @lkoutsofios opinion here? that would help a ton in my view. others should chime in to voice their opinions, of course.

case in point: some harm had already been done in recent months in different distros by ksh2020 displacing ksh93u+ as “the ksh” package due to confusion/unawareness on the side of the package maintainers. specifically this has happened at least in FreeBSD and Debian and OSX. don’t know whether FreeBSD is aware now and will follow debian to again provide both (see below). OSX can’t do that right now due to inability of compiling the old ksh93u+ code (with the nmake based built-system…) and provides only a “ksh” package containing ksh2020 while listing the old ksh93 package as “obsolete and replaced by ksh”. the maintainer there clarified that he would be happy to provide the true ksh93u+ as well if I tell him where to find an easily buildable download .

and in debian it has only been reverted very recently after bringing the problem to the attention of the maintainer there (so that there now is a ksh93u+ package back and in parallel k2020 (which is no problem: if someone likes it better than the real thing: fine, go ahead…).

so it would definitely be beneficial if the “branding” question would be sorted out unambiguously: what is (and what is not) the ATT ksh shell.

overall it seems, ksh2020 should find its own place as a fork (if the project insists on maintaining commit access for krader…, otherwise, possibly, on a branch), get a different landing page (no longer claiming that the visitor is looking at the AST software…) and clearly describe itself as a ksh93v- descendant to be unambiguously distinguished from the ATT ksh93 shell (i.e.93u+) and, possibly, its carefully curated future bug-fix releases. my feeling is: a fork would be the best solution and most accurately reflect what has happened in ksh2020.

I will further discuss with colleagues and come back here if any new aspects/ideas pop up.

react any which inappropriate way you want.

Okay, I’m closing this since the problem is due to an algorithm chosen long ago by the old AST team. Their choice of a recursive algorithm for that pattern is a pretty common choice. It is not an implementation bug. We’re not going to change the algorithm they implemented since doing so would be incredibly risky with respect to backward compatibility.

@jghub I’ll grant that you know more about the behavior of the ksh language than I do (despite my using ksh for more than 15 years). Heck, you might be the leading expert on the planet on the topic of ksh behavior. But you are rude, argumentative, and obnoxious. Please, go away.

Google “if all you have is a hammer.”

this coming from you… so you can be funny (even if only involuntarily).

but let’s leave it at that here: segfault reported. this one of the “111” kind (present in ksh93u+, present in ksh93v-, present in ksh2020). put it on the list.

oh, and maybe count the “111” segfaults and user-facing operational bugs you know of. and than count the bugs in the “011” set (only present in 93v- and 2020 but working in 93u+: I am suspecting this is the largest set) and the :“110” and “010” ones (that would be real fixes in 2020). and the “001” (only buggy in 2020) ones.

A backtrace of the failure shows the stack being exceeded. Here are just the last few lines of the backtrace:

    frame #27214: 0x000000010fc020c5 ksh`parserep + 393
    frame #27215: 0x000000010fc00324 ksh`regnexec_parse + 4989
    frame #27216: 0x000000010fc00d20 ksh`regnexec_parse + 7545
    frame #27217: 0x000000010fbfeced ksh`regnexec + 1125
    frame #27218: 0x000000010fc17633 ksh`strngrpmatch + 387
    frame #27219: 0x000000010fb7d411 ksh`test_strmatch + 157
    frame #27220: 0x000000010fbd9904 ksh`sh_exec + 6164
    frame #27221: 0x000000010fbb48cd ksh`exfile + 1437
    frame #27222: 0x000000010fbb543c ksh`sh_main + 1779
    frame #27223: 0x00007fff654307fd libdyld.dylib`start + 1
    frame #27224: 0x00007fff654307fd libdyld.dylib`start + 1

Note the number of call frames. The three frames from #27214 to #27216 repeat until the stack size limit is exceeded. I would argue this is not a bug in the usual sense of an implementation bug; e.g., a buffer overflow or off by one error. But there is a strong argument to be made that it is a design bug in how the ksh regexp engine handles that pattern using recursion. Especially since, in this case, the shortest match is all that is needed and thus no recursion whatsoever is necessary even if recursion is otherwise required to match other inputs.

Bug reports are great and welcomed. But given your intense interest in a project you’ve repeatedly claimed has been “hijacked” I’d be more impressed if you actually provided an analysis of the bug and, ideally, a fix. If nothing else you could have provided a backtrace of the failure.