Category Automation
Optimizing the Parameters of an Evolutionary Algorithm for Fuzzing and Test Data Generation
Date | 23 March 2020 |
Event | 3rd IEEE Workshop on NEXt level of Test Automation 2020 (NEXTA 2020) |
Location | Porto, Portugal |
Communication interfaces are particularly
challenging to test using automatically generated test
data. The test data sent through the interface must be
”valid enough” to overcome initial sanity checks of the
interface and reach functions deep inside the integrated
software. Machine-readable information about what
data forms ”valid enough” messages is rarely available
to test data generation tools. So instead, we evolve the
messages with an evolutionary algorithm. This enables
efficient fuzz testing for the communication interface
between a satellite and its ground station. In this pa-
per, using an algorithm implementation in our fuzzing
tool DCRTT, we investigate the impact of algorithm
parameter selection on the performance and the possi-
bility of efficient general default parameter values. The
preliminary results promise significant improvements
to automated testing with respect to software security
testing and quality assurance.
Permalink
Automation of Requirements-based Testing
Date | 4 Jun 2019 - 6 Jun 2019 |
Event | DASIA 2019 |
Location | Torremolinos, Spain |
Manual requirements-based testing is time-
consuming: Input data must cover the requirements and
observed output data must be checked for their compatibility
with the requirements. Testcases can also be automatically
generated from test models. However, these models first have to
be established manually. In contrast, the approach to be
presented here uses simpler ways of formalizing requirements
to automatically map test data generated for automatic
robustness testing using massive stimulation to requirements and
to check the results for correctness.
Permalink
Generating Random Telecomand Test Data Using Genetic Algorithms
Date | 29 May 2018 - 31 May 2018 |
Event | DASIA 2018 |
Location | Oxford, United Kingdom |
Generating useful test data is one of the big
challenges in automatic software testing. While random test
data generation is the easiest method, the test inputs generated
by it may fail to exercise the software under test properly if the
internal structure of the data is unknown to the generator and
at the same time relevant for the decisions taken in the code.
Handling of telecommands in space onboard software is one
example where this is the case. We investigate a method of
generating test data for these cases using genetic algorithms.
Permalink
Evaluating Automated Software Verification Tools
Date | 10 April 2018 - 12 April 2018 |
Event | ICST 2018 |
Location | Västerås, Sweden |
Automated software verification tools support devel-
opers in detecting faults that may lead to runtime errors. A fault
in critical software that slips into the field, e.g., into a spacecraft,
may have fatal consequences. However, there is an enormous
variety of free and commercial tools available. Suppliers and
customers of software need to have a clear understanding what
tools suit the needs and expectations in their domain. We selected
six tools (Polyspace, QA C, Klocwork, and others) and applied
them to real-world spacecraft software. We collected reports from
all the tools and manually verified whether they were justified.
In particular, we clocked the time needed to confirm or disprove
each report. The result is a profile of true and false positive
and negative reports for each tool. We investigate questions
regarding effectiveness and efficiency of different tools and their
combinations, what the best tool is, if it makes sense at all to
apply automated software verification to well-tested software, and
whether tools with many or few reports are preferable.
Permalink
Evaluation of Verification Tools Continued: More Tools, More Software, More Aspects
Date | 30 May 2017 - 1 Jun 2017 |
Event | DASIA 2017 |
Location | Gothenburg, Sweden |
In a previous study six software verification tools
have been applied to a representative space software package.
The findings reported by each tool have been compared in
order to derive footprints regarding fault identification. In a
continuation three more tools were applied to the previously
selected application software and to another application
together with two tools previously used in order to broaden the
base of evaluation. More aspects were considered regarding
the evaluation of results: an additional evaluation criterion was
added and a comparison of reported defects with the outcome
of unit tests was performed. Due to a higher degree of
formalization and automation the manual evaluation effort
could be decreased while extending the number of considered
reports and the number of tools. The encountered evaluation
and verification issues are discussed in detail. All results
together shall provide a detailed view on the defect
identification capabilities of the considered tools w.r.t. current
software base. Altogether, the high quality of reports as
obtained in the previous study was not obtained again: in
context of a different set of tools and another (object-oriented)
language a lot of trivial reports were observed.
Permalink
Challenges Regarding Automation of Requirements-based Testing
Date | 30 May 2017 - 1 Jun 2017 |
Event | DASIA 2017 |
Location | Gothenburg, Sweden |
Testing as a method of software verification is
limited in that it can only prove the presence of defects, not
their absence. To be useful, a large number of test cases may
be needed, a strategy that is often in conflict with project
constraints such as available time and funds. Test automation
may be considered as an interesting approach to alleviating
this conflict. However, test automation requires accurate and
computer-accessible information about the system to be tested,
both in terms of the interfaces by which the system is to be
stimulated as well as the desired properties of these interfaces.
Within the FASTII activity (FAST=Flow-optimised Automated
Source-code based Testing) the possibility of deriving this
information from available requirements and design
documents is being investigated. Preliminary results of this
investigation as well as suggestions for future changes in the
process are presented in this paper.
Permalink
Early Results from Characterizing Verification Tools through Coding Error Candidates Reported in Space Flight Software
Date | 10 May 2016 - 12 May 2016 |
Event | DASIA 2016 |
Location | Tallinn, Estonia |
Six software verification tools have been applied to
space flight software and the findings reported by each tool
have been compared in order to derive footprints of the tools
regarding capabilities of fault identification. Currently
available results are provided in this paper: sensitivity and
precision of individual tools and combinations of pairs of tools
out of the set. A reader should bear in mind that the results as
presented here depend on the spectrum of fault types as
present in the reference software and on the configuration of
tools towards real defects and fault types which are of interest
for embedded systems and space flight software.
Permalink
Characterizing Verification Tools through Coding Error Candidates Reported in Space Flight Software
Date | 19 May 2015 - 21 May 2015 |
Event | DASIA 2015 |
Location | Barcelona, Spain |
Mastering the continuously increasing amount of
software requires identification of more efficient strategies for
software verification. Currently, fault coverage is only
indirectly addressed, e.g. by code coverage. The idea as
presented in this paper is to get a better understanding of fault
coverage by a systematic classification of software fault types,
derivation of footprints of verification tools regarding coverage
of such fault types, and recording of required effort. A number
of issues regarding fault identification and classification are
discussed in this context.
Permalink
Automatic Test Data Generation and Model Checking with CHR
Date | 18 Jul 2014 |
Event | CHR 2014 |
Location | Vienna, Austria |
We present an example for application of Constraint Handling
Rules to automated test data generation and model checking in verification
of mission critical software for satellite control.
Permalink
Automated Source-code-based Testing of Object-Oriented Software
Date | 3 June 2014 - 5 June 2014 |
Event | DASIA 2014 |
Location | Warsaw, Poland |
With the advent of languages such as C++ and Java
in mission- and safety-critical space on-board software, new
challenges for testing and specifically automated testing arise.
In this paper we discuss some of these challenges, consequences
and solutions based on an experiment in automated source-
code-based testing for C++.
Permalink
A Case Study on Automated Source-Code-Based Testing Methods
Date | 14 May 2013 - 17 May 2013 |
Event | DASIA 2013 |
Location | Porto, Portugal |
We present results of a case study on a test
generation approach called Flow-optimized Automated
Source-code-based unit Testing (FAST) which generates test
stimuli from information available in the source code, in
particular taken from the detailed software interfaces. This
allows automation of a significant part of testing, ranging from
the test stimuli generation to the generation of the test report.
A huge number of stimuli can be generated exploring the
behaviour of the software under test under nominal and non-
nominal conditions. Symptoms like timeouts, unexpected
termination, run-time exceptions, out-of-range conditions and
missing coverage are applied for defect detection. The goal of
this study was to evaluate the FAST process in context of a real
spacecraft flight software application and to get a feedback on
its scalability regarding larger applications, its sensitivity on
detecting defects in the code, the achievable test coverage, its
compliance with software standards and potential limitations.
We also consider the impact of coding style on suitability for
automated testing. The results confirm that the approach (1)
provides acceptable code coverage results without requiring
manual intervention for test preparation and execution, (2)
raises the probability of activation of exotic fault conditions,
(3) may provide hints on locations in the code where
robustness needs to be verified, and (4) identifies defects not
found before by static analysis and intensive testing
Permalink
Potentials of Constraint-Based Methods in Software Verification and Validation
Date | 14 May 2012 - 16 May 2012 |
Event | DASIA 2012 |
Location | Dubrovnik, Croatia |
We give an overview over the principles of constraint-
based test data generation, discuss its limitations and
potentials and touch some of the domains which may be
interesting to combine with constraint-based testing
techniques. Automated generation of test data is an
example where this technique can be applied and
significantly increase the degree of automation, but it is
not limited to. This paper is intended to give interested
readers a quick entry into the methods and applications
to allow a deeper understanding and an informed verdict
about the actual capabilities and potential future
directions.
Permalink
Integrated Design and Testing of Safety-Critical Real-time Systems in Space
Date | 19 Oct 2011 |
Event | MBTUC 2011 |
Location | Berlin, Germany |
Permalink
Files
Cleared Direct To Target - Approaching the Target System at a Touch via Simulation
Date | 2 Nov 2010 - 4 Nov 2010 |
Event | AirTec 2010 |
Location | Frankfurt am Main, Germany |
This paper describes an improved process for
development of software for critical systems fully
bridging the gap between a specification and an
executable target version by an automaton. The
process covers the broad class of distributed
and/or fault-tolerant and/or real-time systems, and
meets the needs of critical systems. It has its roots
in the space domain.
Permalink
Generic and Extensible Automatic Test Data Generation for Safety Critical Software with CHR
Date | 20 Jul 2010 |
Event | CHR 2010 |
Location | Edinburgh, Scotland |
We present a new method for automatic test data generation
(ATDG) applying to semantically annotated control-flow graphs (CFGs),
covering both ATDG based on source code and assembly or virtual machine
code. The method supports a generic set of test coverage criteria, including
all structural coverage criteria currently in use in industrial software test for
safety critical software.
Several known and new strategies are supported for avoiding infeasible paths,
that is paths in the CFG for which no input exists leading to their execution.
We describe the implementation of the method in CHRv and discuss
difficulties and advantages of CHR in this context.
Permalink
Tool-Guided, Domain-Specific, Systematic Requirements Management
Date | 1 Jun 2010 - 4 Jun 2010 |
Event | DASIA 2010 |
Location | Budapest, Hungary |
The importance of the quality of requirements for
successful execution and completion of a project from a
technical and contractual point of view is being
recognized more and more. Many methods are targeted
to improve the support for collecting requirements
while still focusing on natural language. However, the
ambiguities in the semantics of natural language are the
biggest obstacles towards success. The approach
presented in this paper focuses on the elements of a
domain while keeping the expressiveness of natural
names and terms and introducing clear semantics. This
brings the advantage that immediate verification of the
human-provided inputs is possible, immediate
contributions to validation are available and
inconsistencies can be detected by a tool immediately.
This leads to guidance of an engineer by a tool towards
consistent, complete and correct requirements -
requirements of high quality - and eases maintenance
for the same reasons. As most of the complexity is
handled by the tool due to its good knowledge on the
domain, the approach is scalable towards large
specifications. Several examples of application domains
are described which illustrate the universality and
feasibility of the approach across domain boundaries.
Permalink
Fault Identification Strategies
Date | 26 May 2009 - 29 May 2009 |
Event | DASIA 2009 |
Location | Istanbul, Turkey |
Various strategies for fault identification exist - e.g.
based on formal analysis of code or on testing - of
which each focuses on certain identification aspects and
fault types. This paper characterises the strengths and
weaknesses of methods in theory and practice -
focusing on application-independent identification
strategies, and it suggests strategies to maximise the
number of detected faults while minimising the related
effort. Fault activation conditions are discussed in
detail, resulting in an extended scope on stimulation
needs. In particular, the contribution of automation in
raising the activation probabilities is investigated.
Various examples of fault activation mechanisms and
statistics on fault types vs. identification methods are
provided as observed in practice. An interesting result is
the identification of application-dependent test cases by
application-independent test strategies.
Permalink
Model Transformation in Practice
Date | 29 May 2007 - 1 Jun 2007 |
Event | DASIA 2007 |
Location | Naples, Italy |
The intention of this paper is to highlight the benefits of
model exchange between different tools, methods and
notations on one side, and to identify issues of proper
modelling on the other side which have been detected
during model transformation and code generation from
models.
Permalink
Evaluation of Auto-Test Generation Strategies and Platforms
Date | 29 May 2007 - 1 Jun 2007 |
Event | DASIA 2007 |
Location | Naples, Italy |
As the test effort takes a significant part of the software
development lifecycle, efficient test strategies are a precondition
for reduction of development costs and time.
In this respect two main issues exist: firstly, the tuning
of the test track from test case identification to
evaluation, secondly, the reduction of number of test
cases to be processed and evaluated. Both aspects were
considered in the work presented in this paper.
Permalink
Improving Test Automation by Deterministic Methods in Statistical Testing
Date | 22 May 2006 - 25 May 2006 |
Event | DASIA 2006 |
Location | Berlin, Germany |
This paper was presented at the DASIA 2006 conference in Berlin, Germany.
It lays out principal approaches for tackling the "small target" and "oracle" problems in statistical automatic testing by applying deterministic methods for assistance.
Permalink
Software Diversity by Automation
Date | 30 May 2005 - 2 Jun 2005 |
Event | DASIA 2005 |
Location | Edinburgh, Scotland |
This paper was presented at the DASIA 2005 conference in Edinburgh, Scotland.
It shows the result of an Automated ISVV (AISVV) activity executed on the
Flight Application Software (FAS) of the Autonomous Transfer Vehicle (ATV).
Permalink
Automated Software Production - Quo Vadis
Date | 30 May 2005 - 2 Jun 2005 |
Event | DASIA 2005 |
Location | Edinburgh, Scotland |
Permalink
Automation in the Software Lifecycle, Tailoring ECSS-E-40 for Automatic Code Generation
Date | 7 Feb 2005 - 8 Feb 2005 |
Event | ESA/BSSC Workshop on the Usage of ECSS Software Standards for Space Projects |
Location | Noordwijk, The Netherlands |
Permalink
Product Assurance and Automatic Code Generation - Tailoring ECSS-Q-80 for ACG
Date | 7 Feb 2005 - 8 Feb 2005 |
Event | ESA/BSSC Workshop on the Usage of ECSS Software Standards for Space Projects |
Location | Noordwijk, The Netherlands |
Permalink
Mehr Effizienz durch skalierbare und selbst-adaptierende Softwareentwicklungsprozesse
Date | 4 Nov 2003 - 5 Nov 2003 |
Event | 2. Raumfahrttechnologietage DLR |
Location | Bonn, Germany |
Permalink
Perspektiven der Softwareentwicklung - Probleme und Lösungen
Date | 9 Jul 2003 - 10 Jul 2003 |
Event | Raumfahrttechnologietage DLR |
Location | Bonn, Germany |
Permalink
Avoiding Malfunctions Due To Software Failures by Automation of Software Production and Test
Date | 6 Nov 2002 |
Event | Colloquium: Technology Exchange between Space and Automotive Industry |
Location | Darmstadt, Germany |
This is the presentation as given during the Colloquium: Technology Exchange between Space and Automotive Industry in Darmstadt, Germany (Nov 6th, 2002).
Permalink
Automatische Software Produktion-Was ist möglich?
Date | 15 Oct 2002 |
Event | MicroConsult Praxisforum "Embedded Quality - Neue Dimensionen der QualititÀtssicherung" |
Location | Munich, Germany |
This is the more practice-oriented presentation as given in
Munich (Oct 15, 2002).
If you would like to get an example file package as shown during the
presentation please contact us.
Permalink
Automatische Software Produktion-Was ist möglich?
Date | 10 Oct 2002 |
Event | MicroConsult Praxisforum "Embedded Quality - Neue Dimensionen der QualititÀtssicherung" |
Location | Zurich, Switzerland |
This is the strategy-oriented presentation as given during the MicroConsult Praxis Forum "Embedded Quality - Neue Dimension der QualitÀtssicherung" in Zurich (Oct 10th, 2002).
Permalink
Files
References
Links
Benchmarks on Automated System and Software Generation
Date | 13 May 2002 - 16 May 2002 |
Event | DASIA 2002 |
Location | Dublin, Ireland |
This paper was presented at the DASIA 2002 conference in Dublin, Ireland. It
shows results of benchmarks on projects carried out using ISG/ASaP.
Permalink
Files
Automated Software Production(ISG/ASaP)
The ISG/ASaP approach allows automated system generation from engineering
information and to immediately build an executable system from engineering
level.
This approach has already been successfully applied to provide the
infrastructure and major parts of the application software for the
Material Science Laboratory aboard the International Space Station(ISS).
Permalink
Files
Kosteneinsparung und QualitÀtsverbesserung durch Automation in der System- und Softwareentwicklung(German)
Date | 24 Oct 2001 |
Event | KooperationsForum Computer Hard- und Software |
Location | Berlin, Germany |
This paper describes the potential of ISG/ASaP for cost-savings and quality improvements.
Permalink
Files
Software Development of Embedded Systems by ISG - To Be Faster at Less Costs and Higher Quality
This paper describes the benefits of ISG/ASaP in the context of embedded systems.
Permalink
Files
ISG and Software Quality
The impact on the improvement of software quality by ISG/ASaP
is discussed by this paper.
Permalink
Files
Automation in der Softwareentwicklung - ausfürhbare Progrfamme ohne Programmierung mit ISG (Instantaneous System and Software Generation) erzeugen
Date | 28 Apr 2001 |
Event | Arbeitskreis Studenten und Jungingenieure, VDI |
Location | FH Sigmaringen-Albstadt, Albstadt, Germany |
The presentation covered application of automation in the area of software engineering and included a demo of generation of a distributed realtime system with 16 processes within ca. 40 minutes.
Permalink
Be Faster on The Market at Less Costs by ISG
A comparison of the conventional and the ISG life-cycle and evaluation of cost savings by ISG.
Permalink
Files
Down-scaling Generation, Verification and Validation of Software into the Range of Hours: The Benefits of Complete Formalisation and Automation
This paper discusses the advantages of formalisation and automation of software production and the
challenges yet to master with current operation platforms.
Permalink
Files
ISG and Synchronous Systems
This presentation summarizes the results as obtained by ISG/ASaP in the course
of the ESPRIT project CRISYS (EP 25514). The goal was to evaluate the
robustness in presence of time-jitter and fault-injection for a critical
distributed control system.
Permalink
Files
Links
Instantaneous System and Software Generation: A brief introduction and survey
In this paper BSSE's motivation behind ISG/ASaP is presented and
its relevance to the Capability Maturity Model(CMM) defined by Carnegie-Mellon
Software Engineering Institute. It is shown that by use of
Automated Software Production the upper levels of CMM are reached immediately.
Permalink
Files
An Implementation and Verification Technique for Distributed Systems
Date | 19 Jun 2000 - 23 Jun 2000 |
Event | MOVEP2k: Modelling and Verification of Parallel Processes |
Location | Nantes, France |
Verification and validation of distributed systems as defined by the ISG-approach is
described in this paper.
Permalink
Files
Automated generation of real-time software from datasheet-based inputs: The process model, the platform and the feedback from the MSL project activities
Date | 22 May 2000 - 26 May 2000 |
Event | DASIA 2000 |
Location | Montreal, Canada |
This paper describes how ISG was used to built MSL software.
Permalink
Files
Software Development for the Material Science Laboratory on ISS by Automated Generation of Real-time Software from Datasheet-based Inputs
Date | 22 May 2000 - 26 May 2000 |
Event | DASIA 2000 |
Location | Montreal, Canada |
This paper was presented during the DASIA 2000 conference in Montreal, Canada by
Kayser-Threde.
It describes the organisation of the database from which MSL software was
automatically generated by ISG/ASaP.
Permalink
Files
Built-In Performance and Robustness Engineering Capabilities by a Formalised and Automated Software Development Process
Date | 17 May 2000 |
Event | 1st German Workshop on Performance Engineering (PE2000) |
Location | Darmstadt, Germany |
This paper was presented on the
1st German Workshop on Performance Engineering PE2000 at Darmstadt. It
discusses performance and robustness of software in context of ISG.
The revised version of this paper
"Performance and Robustness Engineering and the Role of Automated Software Development"
is under copyright of Springer Verlag and can be viewed as
LNCS #2047.
Permalink
Files
Links
Procurement of a SDL Model for Behavioural Validation of MSL
This is the final report of the ESA study on behavioural validation of MSL.
Permalink
Files
A Method for the Incremental Software Development and Validation of Complex Distributed Systems
Date | 29 Sep 1999 |
Event | INCOSE meeting |
Location | Bosch telecom, Backnang, Germany |
In the presentation application of incremental development and validation in the area of complex distributed systems was explained.
Permalink
Organizing Incremental, Reusable and Automated Software Development
Date | 17 May 1999 - 21 May 1999 |
Event | DASIA'99 |
Location | Lisbon, Portugal |
Description of an approach for incremental software development and
validation applying automated generation of an executable system
from a minimum of system engineering information.
Permalink
Files
Verification with SDL - Some Hints about How to reduce size of state space
This paper gives advice how to master state explosion.
Permalink
Files
Tuning Development of Distributed Real-Time Systems with SDL and MSC: Current Experience and Future Issues
Date | 23 Sep 1997 - 26 Sep 1997 |
Event | SDL'97 Forum |
Location | Evry, France |
Permalink
Files
References
Links
Tuning Development of Distributed Real-Time Systems with SDL and MSC: Current Experience and Future Issues
Date | 23 Sep 1997 - 26 Sep 1997 |
Event | SDL'97 conference |
Location | Evry, France |
This paper was published by
Elsevier Science Publishers B.V., ISBN: 0-444-82816-8
Permalink
Files
References
Experience with Validation by Simulation, Automated Code Generation and Integration
Date | 26 May 1997 - 29 May 1997 |
Event | DASIA'97 |
Location | Sevilla, Spain |
The presentation included an online demonstration on a Sparc Workstation and a PC laptop.
Permalink
Files
From CASE to CIVE: A Future Challenge!
Date | 20 May 1996 - 23 May 1996 |
Event | DASIA'96 |
Location | Rome, Italy |
This paper was presented during the DASIA'96 conference, May 20th - 23th 1996
in Rome.
Permalink
Files
Use of an extended SDL Environment for Specification and Design of On-Board Operations
Date | 28 Nov 1995 - 30 Nov 1995 |
Event | Systems Engineering Workshop |
Location | ESTEC, Noordwijk, The Netherlands |
This paper was presented during the Systems Engineering Workshop,
November 28th - 30th 1995 at ESTEC, Noordwijk.
Permalink
Files
Links
A Random-Testing Environment for Ada Programs
Date | Nov 1993 |
Event | Eurospace Symposium "Ada in Aerospace" |
Permalink