Wednesday, July 24, 2013

Yocto: mature or not?

This is a bit of fun spoiling post, but sometimes we need to become constructively critical in order to improve the situation. As you may already know, Yocto has been advertised as the relatively new technology for creating custom distributions for you. Although, people commonly use Poky, Arago, and so forth ready-made distributions on top of Open Embedded.

Yocto Project

Cross-compilation


So, here is the critical issue for embedded developers using the cross-toolchain provided by Code Sourcery why I personally think that Yocto might not be usable for the moment for commercial projects where reliability is a main concern.

In short: it does not work if you would like to build the distribution (for instance stock Poky) for your embedded board, like ARM with the sourcery toolchain.

What makes me worry a bit more, this cross-toolchain environment was working in the first half of 2012 yielding some regression in here. That opens up new concerns about QA, right? It is fundamentally broken and can be reproduced easily as acknowledged in the bug report above, yet the changes got in.

One could suggest to revert back to the very old and unsupported version. Well, it is unsupported first of all, so you cannot reliable base a commercial product on top of it. More importantly, that version has a lot more issues. I have tried to back port fixes to that release from master, but I have not had so much fun. I experienced many merge conflicts and so forth.

Cross compilation

Unsupported Archlinux

It is a bit disappointing that such a common distribution as Arch is not supported. Please do not write me that they do not enough manpower. I fully understand and appreciate that. Yet, it means Archlinux developers are in trouble.

Archlinux


Unsupported releases

They, kinda, support two releases back. They have a time based release cycle. One new release comes in every half a year. This brings us to the point that they only support your selected software version up to maximum one year. That is not so comfortable when you need a reliable product. I just faced the issue yesterday that there are fixes in old release branches, but no one has cared to release them for more than half a year. They will probably not be released either. I am now referring to the denzil version coming from April in 2012. So, it is not quite like the Linux kernel or Ubuntu LTS.

Long Term Support (LTS)

Error reporting

I think this is also a hard sell in Yocto as of now. There are issues that I could not simply figure out on my own as a newcomer without quite a bit of time with analyzing, especially when the public documentation is not up to the task either. There are several issues which made me spend a few days just to understand them, and that what is going on underneath.

The first bogus error reporting is about bblayers.conf file mismatch based on the generated file and the sample available. It was even hard for others to provide some help on IRC who have been experienced with the project. Here you can read the details about it.

Then, there is the issue of mismatching cross-compilation toolchain and MACHINE configuration. In my case, I would like to use the arm gnueabi toolchain from gcc for my omap board, particularly for the arm core. All fine, the external sourcery toolchain is setup, but the default config file generated for the build comes with "qemux86". When you try to generate the distribution, you will get weird toolchain issues about x86. I have been surprised because I explicitly specified the toolchain what to build with. Having spent 1-2 days with trying to understand the situation, it boils down to that the default MACHINE config generated, screwed this up. It would have been so much easier if it is reported up front with a warning that the toolchain and machine configurations are mismatching. Here you can find the bugreport about that one.

There are more to it, albeit the last, but not least I would like to mention here is the one when you get tab/space issues when migrating from an older poky version, or you just get those for some reason. Reporting those issues should be easy, shouldn't it? Theoretically yes, but in practice, all you will get a file having issues tab and space issues. I did not quite get the point what the issue is after checking the offending file for quite a while. It turned out that the real problem is in the include file which it was requiring. Again, a newbie would spend a lot of time with it without external and professional support. I would even go further than reporting the right file as in: it would be nice to get a report for the location as close as possible. That is, a function name, perhaps a list of offending line numbers, and so forth. Here you can find the relevant bugreport for this.

Please note that, I have been told for some of these that it is not an easy fix as the parser design is not prepared for such use cases. It unfortunately means, it is not just about adding a minor feature, but basically revamping the parser.

Error messages

Documentation

There are some issues here as well I hit. I had a DISTRO_FEATURE entry "--disable-zlib" that I inherited. I was looking into it, but I have not seen anything like that in the reference manual. I guessed respestively that, it is not a proper syntax, so I can remove it. Right, you would imagine it could be as simply as that?

The problem is that, yes, theoretically, but in practice: I have seen other undocumented DISTRO_FEATURES like "opengl". I opened up a bugreport for that, but the problem is that, once you find a feature not documented, how will you know what to do with another not represented. Is that a mistake, undocumented, or what exactly? :)

I would not like to bore the reader, so I will only give one more example in here. I have been told that external toolchains should work, yet, it is not documented in the released documentation properly. Even in the development version of the next generation documentation, it is a bit of sloppy. You get the documentation of 2-3 variables, but you do not have any concrete example at hand. Then, I have been told to use the meta-toolchain/sourcery layer. As for the latter, I do not even find any documentation that could be useful for me on github from Mentor Graphics. I created a bugreport for that one, too.

Dr. Documentation

Conclusion

I am objectively pessimistic with the current state as an Arch user who is looking for a reliable product in an embedded environment with cross-compilation. I have spent a couple of days to experiment with all this, and I just wanted to let you the limitations so that you do not need to go through the same issues. Oh, and please do not write that "It works for me" because yes, I heard people using git master or not even that with supported distributions, and very simple setups where cross-toolchain is not necessary, et cetera. I am now referring to Arch, embedded, and so forth scenarios. I of course appreciate the Yocto people's work a lot, and I hope they can put more effort into this project to address those issues in the, hopefully near, future.

Conclusion

7 comments:

  1. Hi.

    I'm quite a begginner with the embedded world, but let me comment on some things. First, how would Arch be supported? Wikipedia tells me is a rolling release distro, so how can that be supported something that is always moving?

    If you had the chance to attempt to install some of these tools on Arch, consider yourself lucky. I have an Ubuntu 10.04 virtual machine because I depend on an ancient LTIB installation provided by Freescale. Anything newer seems to displease an already fragile tool like LTIB. If you could do anything on a Debian chroot, I would totally change my place with you. :-)

    And besides, what do you think could be the alternatives? My first contact with embedded was such ancient LTIB, and the only tools that I hear that people like are ScratchBox2, BuildRoot and Yocto. And from what I've seen (again, I'm a newbie with this stuff), is easier if you surrender and use the same tool that the manufacturer suggests.

    Greetings.

    ReplyDelete
  2. Hi suy,

    thank you for your questions.

    1) There have been projects supporting arch. I believe it is the matter of focus. It is not hard to maintain it according to my experience. It is sometimes even simpler because they integrate bugfix and feature releases earlier than other distributions. That can help with many things, including the elimination of nasty workarounds around, et cetera.

    2) Not sure about how much luck we need. :)

    Yocto already runs on arch with master if I do not use either the cross-toolchain or the external Code Sourcery.

    This issue comes with the CS toolchain, but for desktop or perhaps even with internal toolchain, it could possibly work.

    3) I am currently using a Debian Wheezy (7.0) chroot, yes, but that also has its own quirks. However, that probably requires less expertise to solve than the Yocto or Mentor Graphics specific issues. In the meantime, I also devotee my spare time to collaborate with the Mentor Graphics and Intel employees to solve the critical Arch issues for once and all.

    4) I think one alternative is using a ready-made distribution, or just using a supported OS in chroot or virtualization. Luckily enough, at least debian stable is supported. The Mer Project is using scratchbox2 and OBS, but I prefer the Yocto/OE flavor, personally. I have never used buildroot myself, but I have read that it is not so much customizable, albeit I might be totally off there.

    ReplyDelete
  3. I wouldn't expect ArchLinux developers to contribute to upstream software. Most of them are just package maintainers with no background in software development. You can tell by how most bugs are closed with "go report it upstream". There are some ArchLinux developers who can code software but they are simply busy.

    ReplyDelete
  4. This blog post is not about the Archlinux developer community, but the Yocto project itself.

    Off topic, but Archlinux developers sometimes contribute to upstream. I would like to give the credit for them when having done such well-done jobs.

    ReplyDelete
  5. FWIW, I once made a contribution to util-linux because of something that broke in Arch (and would have broken in all the other distributions once they upgraded). I've known others who have done the same, possibly for Yocto as well.

    If you're not comfortable contributing fixes upstream, you shouldn't even be *using* Arch, much less developing it.

    That said, I gave up trying to build Dylan under Arch several months ago. I hack on Yocto as part of my job, and frankly, I didn't have the time for it. Since then I've used debootstrap to assemble an Ubuntu 12.04 chroot, then I do all my building via schroot. It works so marvelously that I don't really feel any need to do "native" builds anymore. Furthermore I think there are strong halo benefits to using chroots for these sorts of builds in general: as a developer, it's extremely useful to verify that something builds under several different distributions, and VM overhead (even using KVM) is somewhat high.

    *None* of that is Yocto's fault. The two biggest issues I ran into that were "Arch-related" were 1) texinfo-5.0 horribly broke binutils, 2) gcc-4.8 ICE'd on the gcc-4.7 build. Both were ultimately GNU's fault, not Arch's.

    ReplyDelete
  6. > If you're not comfortable contributing fixes upstream, you shouldn't even be *using* Arch, much less developing it.

    I reported 30-50 bugreports by now. Accordingly, not sure what you mean by "being uncomfortable with contribution". I even had several patches by now.

    It is not about "being uncomfortable with contribution". It is just that Arch is a widely used distribution. You may probably already know that if you use Arch as you claim.

    Actually, Dylan builds just fine on Arch, thanks to a few contributors. The only thing you need to do is to backport a couple of patches.

    debootstrap is pretty tricky to be used due to their shared memory processing with python. I created a bugreport for that. It is not as simple as it seems. I would suggest "systemd-nspawn" in the future which will resolve the issues for you transparently.

    I do not think that a developer needs to verify any simple changes on many distributions. That is what CIs and their reports are for.

    Sure, it is Yocto's "fault" as you write. I would rather say trade or decision that they do not support Arch properly.

    As for 1-2) that you are mentioning here, they were fixed quite a while ago by Arch people. I am not sure what you mean by they are not arch related. I mean, why would you like to prove that? It is obvious. However, Arch is a bleeding edge distribution, and such issues will come up on this distribution, and for that reason, they will be fixed by Arch contributors.

    Let us not try to pretend it is not Yocto's trade that they decided not to support this widely used distribution, officially.

    ReplyDelete
  7. Hmm, the situation has changed a lot lately. I would like to express my gratitude to Scott about that. He has made an excellent job on that front!

    ReplyDelete