# toolbox
**Repository Path**: liwen_test_sync_group/toolbox
## Basic Information
- **Project Name**: toolbox
- **Description**: Toolbox - go utility library
- **Primary Language**: Unknown
- **License**: Apache-2.0
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No
## Statistics
- **Stars**: 0
- **Forks**: 0
- **Created**: 2020-12-03
- **Last Updated**: 2026-01-30
## Categories & Tags
**Categories**: Uncategorized
**Tags**: None
## README
# Toolbox - go utility library
[](https://goreportcard.com/report/github.com/viant/toolbox)
[](https://godoc.org/github.com/viant/toolbox)
This library is compatible with Go 1.8+
Please refer to [`CHANGELOG.md`](CHANGELOG.md) if you encounter breaking changes.
- [Motivation](#Motivation)
- [Collection Utilities](#Collection-Utilities)
- [Converter && Conversion Utilities](#Conversion-Utilities)
- [Struct Utilities](#Struct-Utilities)
- [Function Utilities](#Function-Utilities)
- [Time Utilities](#TimeUtilities)
- [Storage API](#storage)
- [Data substitution](data/)
- [Text Utilities](text/)
- [ServiceRouter](#ServiceRouter)
- [Decoder and Encoder](#DecoderandEncoder)
- [Logger](#Logger)
- [BatchLimiter](#BatchLimiter)
- [AST Based FileSetInfo](#ast-based-filesetinfo)
- [License](#License)
- [Credits and Acknowledgements](#Credits-and-Acknowledgements)
## Motivation
This library was developed as part of [Datastore Connectivity](https://github.com/viant/dsc/) and Testibility libraries: ([Assertly](https://github.com/viant/assertly), [Datastore testing](https://github.com/viant/dsunit/), [End to end testing](https://github.com/viant/endly/))
as a way to share utilities, and other abstractions that may be useful in other projects.
### Collection Utilities
#### Iterator
Example
```go
slice := []string{"a", "z", "c"}
iterator := toolbox.NewSliceIterator(slice)
value := ""
for iterator.HasNext() {
iterator.Next(&value)
...
}
```
#### Slice utilities
The following methods work on **any slice type.**
**ProcessSlice**
Example
```go
var aSlice interface{}
toolbox.ProcessSlice(aSlice, func(item interface{}) bool {
...
return true //to continue to next element return true
})
```
**ProcessSliceWithIndex**
Example:
```go
var aSlice interface{}
toolbox.ProcessSlice(aSlice, func(index int, item interface{}) bool {
...
return true //to continue to next element return true
})
```
**IndexSlice**
Example:
```go
type Foo struct{
id int
name string
}
var aSlice = []Foo{ Foo{1, "A"}, Foo{2, "B"} }
var indexedMap = make(map[int]Foo)
toolbox.IndexSlice(aSlice, indexedMap, func(foo Foo) int {
return foo.id
})
```
**CopySliceElements**
Example:
```go
source := []interface{}{
"abc", "def", "cyz",
}
var target = make([]string, 0)
toolbox.CopySliceElements(source, &target)
```
**FilterSliceElements**
Example:
```go
source := []interface{}{
"abc", "def", "cyz","adc",
}
var target = make([]string, 0)
toolbox.FilterSliceElements(source, func(item string) bool {
return strings.HasPrefix(item, "a") //this matches any elements starting with a
}, &target)
```
**HasSliceAnyElements**
Example:
```go
source := []interface{}{
"abc", "def", "cyz","adc",
}
toolbox.HasSliceAnyElements(source, "cyz")
```
**SliceToMap**
Example:
```go
var source = []Foo{ Foo{1, "A"}, Foo{2, "B"} }
var target = make(map[int]string)
toolbox.SliceToMap(source, target, func(foo Foo) int {
return foo.id
},
func(foo Foo) string {
return foo.name
})
```
**TransformSlice**
Example:
```go
type Product struct{ vendor, name string }
products := []Product{
Product{"Vendor1", "Product1"},
Product{"Vendor2", "Product2"},
Product{"Vendor1", "Product3"},
Product{"Vendor1", "Product4"},
}
var vendors=make([]string, 0)
toolbox.TransformSlice(products, &vendors, func(product Product) string {
return product.vendor
})
```
#### Map utilities
**ProcessMap**
The following methods work on **any map type.**
Example:
```go
var aMap interface{}
toolbox.ProcessMap(aMap, func(key, value interface{}) bool {
...
return true //to continue to next element return true
})
```
**CopyMapEntries**
Example:
```go
type Foo struct{id int;name string}
source := map[interface{}]interface{} {
1: Foo{1, "A"},
2: Foo{2, "B"},
}
var target = make (map[int]Foo)
toolbox.CopyMapEntries(source, target)
```
**MapKeysToSlice**
Example:
```go
aMap := map[string]int {
"abc":1,
"efg":2,
}
var keys = make([]string, 0)
toolbox.MapKeysToSlice(aMap, &keys)
```
**GroupSliceElements**
Example:
```go
type Product struct{vendor,name string}
products := []Product{
Product{"Vendor1", "Product1"},
Product{"Vendor2", "Product2"},
Product{"Vendor1", "Product3"},
Product{"Vendor1", "Product4"},
}
productsByVendor := make(map[string][]Product)
toolbox.GroupSliceElements(products, productsByVendor, func(product Product) string {
return product.vendor
})
```
**SliceToMultimap**
```go
type Product struct {
vendor, name string
productId int
}
products := []Product{
Product{"Vendor1", "Product1", 1},
Product{"Vendor2", "Product2", 2},
Product{"Vendor1", "Product3", 3},
Product{"Vendor1", "Product4", 4},
}
productsByVendor := make(map[string][]int)
toolbox.SliceToMultimap(products, productsByVendor, func(product Product) string {
return product.vendor
},
func(product Product) int {
return product.productId
})
```
### Converter && Conversion Utilities
Converter transforms, data between any compatible or incompatible data type including struct/basicType/map/slice/interface{}
On top of that it supports custom tag to map field to target data type (i.e map)
```go
myStruct := //some struct ...
myMap := make(map[string]interface{})
converter := NewConverter(dateLayout, keyTag)
err = converter.AssignConverted(&myMap, myStruct)
err = converter.AssignConverted(myStruct, myMap)
```
### Struct Utilities
**ScanStructMethods**
Scan struct methods
```go
service := New()
err = toolbox.ScanStructMethods(service, 1, func(method reflect.Method) error {
fmt.Printf("%v\n", method.Name)
return nil
})
```
**ProcessStruct**
Scan struct fields
```go
service := New()
err = toolbox.ProcessStruct(service,
func(field reflect.StructField, value reflect.Value) error {
fmt.Print(field.Type.Name)
return nil
})
```
### Function Utilities
### Time Utilities
**DateFormatToLayout**
Java date format style to go date layout conversion.
```go
dateLaout := toolbox.DateFormatToLayout("yyyy-MM-dd hh:mm:ss z")
timeValue, err := time.Parse(dateLaout, "2016-02-22 12:32:01 UTC")
```
**TimeAt**
Provide dynamic semantic for creating time object
```go
tomorrow, err = TimeAt("tomorrow")//tomorrow in local timezone
timeInUTC, err := TimeAt("2 days ago in UTC") //or 2DayAgoInUTC
yesterdayUTC, err := TimeAt("yesterdayInUTC")//yesterday in UTC
hoursAhead, err := TimeAt("50 hours ahead")
```
**TimeDiff**
Provide dynamic semantic for creating time object based on time dif
```go
lastyear, _ := time.Parse(DateFormatToLayout("yyyy-MM-dd"), "2017-01-01")
ts1, e := TimeDiff(lastyear, "50 hours earlier")
ts2, e := TimeDiff(lastyear, "3 days before in Poland")
```
**DayElapsed**
```go
t0, _ := time.Parse(DateFormatToLayout("yyyy-MM-dd hh:mm:ss"), "2017-01-01 12:00:00")
dayElapsedInT0, err := ElapsedDay(t0) //0.5
```
**ElapsedToday**
```go
elapscedInLocalTz, err := ElapsedTodayInPct("")
elapscedInUTC, err := ElapsedToday("UTC")
```
**RemainingToday**
```go
elapscedInLocalTz, err := RemainingTodayInPct("")
elapscedInUTC, err := RemainingToday("UTC")
```
**AtTime**
```go
atTime := &AtTime{
WeekDay: "*",
Hour: "*",
Minute: "10,30",
}
//returns the nearest future time for xx:10 or xx:30
nextScheduleTime := atTime.Next(time.Now)
```
## Storage
[Storage API](storage/README.md) provides unified way of accessing local or remote storage system.
This API has been deprecated, please consider using [Abstract Storage](https://github.com/viant/afs)
**Example**
```go
import (
"github.com/viant/toolbox/storage"
_ "github.com/viant/toolbox/storage/gs"
)
destinationURL := "gs://myBucket/set1/content.gz"
destinationCredentialFile = "gs-secret.json"
storageService, err := storage.NewServiceForURL(destinationURL, destinationCredentialFile)
```
### Text utilities
**Text Case Format**
You can format with [format.Case](format/case.go).Format(text, destCase)
```go
formatted := format.CaseLowerUnderscore.Format(text, format.CaseLowerCamel)
```
### Tokenizer
### ServiceRouter
This ServiceRouter provides simple WebService Endpoint abstractin and RESET Client utilities.
Take as example of a ReverseService defined as follow
```go
type ReverseService interface {
Reverse(values []int) []int
}
type reverseService struct{}
func (r *reverseService) Reverse(values []int) []int {
var result = make([]int, 0)
for i := len(values) - 1; i >= 0; i-- {
result = append(result, values[i])
}
return result
}
```
In order to define Endpoint for this service, define a server, a router with the service routes;
```qo
type Server struct {
service ReverseService
port string
}
func (s *Server) Start() {
router := toolbox.NewServiceRouter(
toolbox.ServiceRouting{
HTTPMethod: "GET",
URI: "/v1/reverse/{ids}",
Handler: s.service.Reverse,
Parameters: []string{"ids"},
},
toolbox.ServiceRouting{
HTTPMethod: "POST",
URI: "/v1/reverse/",
Handler: s.service.Reverse,
Parameters: []string{"ids"},
})
http.HandleFunc("/v1/", func(writer http.ResponseWriter, reader *http.Request) {
err := router.Route(writer, reader)
if err != nil {
response.WriteHeader(http.StatusInternalServerError)
}
})
fmt.Printf("Started test server on port %v\n", port)
log.Fatal(http.ListenAndServe(":"+port, nil))
}
```
**ServiceRouting** parameters define handler parameters that can be extracted from URI, QueryString, or from Post Body (json payload)
In addition two special parameter names are supported: @httpRequest, @httpResponseWriter to pass in request, and response object respectively.
The REST client utility invoking our reverse service may look as follow
```go
var result = make([]int, 0)
err := toolbox.RouteToService("get", "http://127.0.0.1:8082/v1/reverse/1,7,3", nil, &result)
//...
err := toolbox.RouteToService("post", "http://127.0.0.1:8082/v1/reverse/", []int{1, 7, 3}, &result)
```
By default a service router uses reflection to call the matched routes handler, it is possible to avoid reflection overhead by providing the custom handler invoker.
```go
var ReverseInvoker = func(serviceRouting *toolbox.ServiceRouting, request *http.Request, response http.ResponseWriter, uriParameters map[string]interface{}) error {
var function = serviceRouting.Handler.(func(values []int) []int)
idsParam := uriParameters["ids"]
ids := idsParam.([]string)
values := make([]int, 0)
for _, item := range ids {
values = append(values, toolbox.AsInt(item))
}
var result = function(values)
err := toolbox.WriteServiceRoutingResponse(response, request, serviceRouting, result)
if err != nil {
return err
}
return nil
}
//...
router := toolbox.NewServiceRouter(
toolbox.ServiceRouting{
HTTPMethod: "GET",
URI: "/v1/reverse/{ids}",
Handler: s.service.Reverse,
Parameters: []string{"ids"},
HandlerInvoker: ReverseInvoker,
})
//...
```
### Decoder and Encoder
#### Decoder
This library defines DecoderFactory interface to delegate decoder creation,
This library comes with standard JSON and UnMarshaler (protobuf) factory implementation.
Example
```go
factory :=toolbox.NewJsonDecoderFactory()
....
decoder := factory.Create(reader)
foo := &Foo{}
err = decoder.Decode(foo)
marshalerFactory := toolbox.NewUnMarshalerDecoderFactory()
decoder := marshalerFactory.Create(reader)
foo := &Foo{}
err = decoder.Decode(foo)
```
#### Encoder
This library defines EncoderFactory interface to delegate encoder creation,
This library comes with standard JSON and Marshaler (protobuf) factory implementation.
Example
```go
factory :=toolbox.NewJsonEncoderFactory()
....
buffer := new(bytes.Buffer)
decoder := factory.Create(buffer)
err = decoder.Encode(foo)
marshalerFactory := toolbox.NewMarshalerEncoderFactory()
decoder := marshalerFactory.Create(buffer)
err = decoder.Encode(foo)
```
### Logger
This library provides a file logger implementation that optimizes writes.
Log messages are queues until max queue size or flush frequency are met.
On top of that Ctrl-C also forces immediate log messages flush to disk.
File template support java style time format to manage rotation on the file name level.
```go
logger, err := toolbox.NewFileLogger(toolbox.FileLoggerConfig{
LogType: "test",
FileTemplate: "/tmp/test[yyyyMMdd-hhmm].log",
QueueFlashCount: 250,
MaxQueueSize: 500,
FlushFrequencyInMs: 2000,
MaxIddleTimeInSec: 1,
}, toolbox.FileLoggerConfig{
LogType: "transaction",
FileTemplate: "/tmp/transaction[yyyyMMdd-hhmm].log",
QueueFlashCount: 250,
MaxQueueSize: 500,
FlushFrequencyInMs:2000,
MaxIddleTimeInSec: 1,
},
)
logger.Log(&toolbox.LogMessage{
MessageType: "test",
Message: message
})
logger.Log(&toolbox.LogMessage{
MessageType: "transaction",
Message: message
})
```
### BatchLimiter
This library provides a batch limiter, that enables controling number of active go routines.
```go
var tasks []*Task
var batchSize = 4
limiter:= toolbox.NewBatchLimiter(batchSize, len(tasks))
for i, _ := range tasks {
go func(task *Task) {
limiter.Acquire()
defer limiter.Done()
task.Run();
}(tasks[i])
}
limiter.Wait()
```
### AST Based FileSetInfo
```go
pkgPath := ""
source := path.Join(pkgPath)
filesetInfo, err :=toolbox.NewFileSetInfo(source)
myType := fileSetInfo.Type("MyType")
fields := myType.Fields()
method := myType.Receivers
```
## GoCover
[](https://gocover.io/github.com/viant/toolbox)
## License
The source code is made available under the terms of the Apache License, Version 2, as stated in the file `LICENSE`.
Individual files may be made available under their own specific license,
all compatible with Apache License, Version 2. Please see individual files for details.
## Credits and Acknowledgements
**Library Author:** Adrian Witas
**Contributors:**