symfony: Thoughts about releases and bugs
Hi everyone, I write this issue because I’m kind of bothered and suprised about how some bugs and BC breaks are easily introduced on almost each version of Symfony, especially on the form component.
- 2.7 was released with major bugs in the form component.
- In 2.8 I encountered a bug on CollectionType about
entry_optionsoption replacingoptionsoption but the prototype was still created from theoptionsoption. - 3.0.3 introduced a bug in the choice list. The choice value was not pre set anymore from existing form data. I found an opened issue and PR fixing it so I waited for the 3.0.4.
- 3.0.4 fixed the choice list issue and boom, another bug, this time on the NumberType which doesn’t cast its value to float. So all my float validation breaks. There is already a PR to fix it, but one more time I have to wait for the next version or install the dev master version.
And there are only some of them I can recall. I don’t remember the last Symfony version I used without this kind of annoying and obvious bugs.
I’m not questioning the awesome work that Symfony contributors make, I know many of them contribute beside their own job. I know it’s not an easy task to manage all that stuff. But my point is : I think there is a lack of checks and tests before a tag is released. These bugs are not edge cases, they concern basic functionalities of the framework used in a large amount of Symfony based applications and are easily detected.
Am I the only one to think that way ? Is the framework functionnaly tested somehow before each release ? What do you think about enforcing the tests on basic Symfony functionalities ? Could it be a good idea to write functional tests like Symfony WebTestCase or behat features covering basics use case with controllers, forms, validations, … ?
About this issue
- Original URL
- State: closed
- Created 8 years ago
- Reactions: 3
- Comments: 19 (18 by maintainers)
I think the hard part is to come up with good test cases. The truth is that the Form component is one of the biggest parts of the framework that offers a huge set of features (even allowing a lot of edge cases we maybe didn’t think about when implementing new features and fixing bugs). The good news is that IIRC issues being fixed once haven’t been introduced again as we always added new test cases when fixing bugs. Though adding additional test cases (best based on real use cases from existing application) would be a huge benefit for the component imo.
In fact the symfony-standard has already doctrine bundle as dependency, you’re right guys, concerning form and security we just need a few controllers and services to have yet a good set of tests.
So why not use
symfony-demo?So because not enough people test the framework before the release, the tag if considered as enough stable to be released ? I think it’s a bit easy to just let people testing. As I said, many of the bugs I encountered aren’t edge cases, a simple form with basic fields is enough to find them. It makes me think that some development or fix are made and released without being tested in real condition at all. I don’t blame anyone, I just wanted to start a discussion about it and find a way to detect easily bugs and regressions to make the framework better.
Well, I’m thinking to start a dummy project with these kind of tests. If some people are interested, it will be on GitHub. Thanks for your feed back.
@xabbuh @stof you are right … but this is not only about bugs. If we had those tests, we could check performance regressions too. So far we only do micro-benchmarks, but we don’t know if the overall Symfony performance is improving/degrading when introducing a new feature.
RubyOnRails for instance has the excellent https://rubybench.org/benchmarks which benchmarks every single commit and release.
and regarding changes in big refactorings like the 2.7 one, a way to avoid such issues in stable release is to migrate projects to 2.7 before the release, during our beta phase (without deploying the updated branch to production of course). The more devs are doing it, the more likely bugs can be fixed before the stable release. Note that I’m guilty myself for that, as I migrated my work project to Symfony 2.7 only after a few patch releases (which is why I contributed many form bug fixes in 2.7.4 or 2.7.5 btw).
@maximecolin thanks for initiating this discussion. It takes some courage to openly criticize a popular Open Source project … but that’s exactly what we need to keep improving. Thank you!
To my understanding those kind of tests don’t exist (and it’s true that they could have prevented the issues you mentioned).
I like your proposal to create those tests as part of our QA workflow. You have my 👍